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Aba  tract 

The  main  focua  of  thia  theaia  waa  to  determine  if 
expoaing  an  Air  Force  ayatem  acquiaition  midlevel  program 
manager  to  a  computer  aided  quantitative  decision  support 
technique  in  an  academic  environment,  effected  the  frequency 
that  the  manager  used  quantitative  decision  support 
techniques  in  actual  program  and  project  problems. 

A  quantitative  decision  support  technique,  decision 
tree  analysis,  was  taught  in  the  Intermediate  Program 
Management  (SYS-400)  Professional  Continuing  Education  (PCE) 
course  at  Wr Ight-Patterson  AFB ,  OH.  Before  this  research 
began,  the  classes  were  doing  the  decision  tree  calculations 
without  the  aid  of  a  computer.  This  study  focused  on 
whether  modifying  the  curriculum  to  teach  the  use  of  PC 
baaed  decision  support  system  (DSS)  decision  tree 
computations  to  program  managers  would  affect  the  frequency 
they  used  quantitative  techniques  In  addressing  actual 
program  and  project  problems.  The  experiment  waa  a 
posttest-only  quaa 1 -exper iment  design  with  nonequivalent 
groups.  This  study  did  not  try  to  discover  if  other 
teaching  techniques  or  other  situations  will  alter  the 
frequency  of  quantitative  technique  use. 

Three  months  after  course  completion,  a  after-course 
survey  was  sent  to  each  course  graduate.  The  survey 
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measured  the  graduate’s  familiarity  and  frequency  of  using 
several  quantitative  decision  support  techniques  to  measure 
the  effect  of  the  classroom  instruction.  Demographic  and 
responsibility  were  collected.  This  Information  was  used  to 
identify  possible  effects,  other  than  the  treatment  effect, 
obscuring  the  analysis. 

This  study  measured  an  increase  (in  the  treatment 
group)  in  some  of  the  technique  familiarity,  use, 
computerization  benefit,  computer  program  availability  and 
use  opportunity.  The  majority  of  program  managers  are 
familiar  with  several  quantitative  decision  support 
techniques.  However,  very  few  managers  use  those  techniques 
regularly . 
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FREQUENCY  OF  QUANTITATIVE  DECISION  SUPPORT  TECHNIQUES 
USED  BY  AIR  FORCE  SYSTEM  ACQUISITION  MANAGERS 


I .  Introduction 

General  lasue 

In  today’s  highly  complex  and  volatile  environment,  the 
program  manager  for  Department  of  Defense  (DoD)  weapon 
system  acquisitions  must  be  able  to  make  intelligent 
decisions  that  will  effect  the  United  States’  national 
interests  (24;  165).  A  program  manager  must  be  able  to 
coordinate  and  integrate  the  actions  and  resources  of 
several  different  organizations  as  well  as  their  own  system 
program  office  (SPO)  (14:46).  The  program  manager  must 
decide  how  to  minimize  the  conflicting  goals  of  the 
project’s  performance,  schedule  and  cost  criteria.  Also, 
pressure  is  exerted  for  the  project  to  meet  suppor tabi 1 i ty 
and  reliability  goals  and  various  other  criteria.  There  are 
constant  demands  for  program  manager’s  time  and  decisions. 

Program  managers  are  constantly  evaluating  different 
trade-offs  that  effect  the  .system  under  development.  They 
frequently  ask  such  questions  as  'which  choice  is  better  for 
the  entire  life  of  the  system?',  'which  trade-off  will  bring 
better  performance?'  or  'what  effect  do  these  choices  have 
on  the  current  schedule  and  cost  projections?'.  However, 
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decision  analysis  assistance  in  answering  these  questions  is 
not  always  readily  available  to  the  managers  (20:151). 

Management  Science  provides  several  quantitative 
decision  support  technique  tools  to  the  program  managers  to 
aid  their  decision  making.  The  availability  oi  personal 
computers  (PC)  and  decision  support  systems  (DSS)  make  the 
information  gained  from  these  techniques  immediately 
available  to  the  manager.  There  are  numerous  advantages  for 
using  a  PC  in  a  program  manager’s  office  (30:33)  Through 
the  use  of  PC-based  decision  support  techniques,  managers 
gain  Immediate  feedback  to  questions  and  "What  If?" 
scenarios.  Just  as  a  hammer  cannot  drive  a  nail  without  the 
carpenter,  microcomputers  can  help  managers  do  their  work 
(5:2) ,  but  are  rarely  used  by  the  managers  (28:77) . 

DSS  in  support  of  battlefield  commanders  demonstrated 
the  successful  application  of  DSS  tools  (POST-PAWS,  FLAPS, 
TERPES ,  etc.)  (50:2-5  -  2-11).  However,  acquisition  program 
managers  lack  similar  support  and  tend  to  rely  heavily  on 
'back  of  the  envelope"  or  "seat  of  the  pants*  decision 
making  (6:1).  This  method  of  solving  modern  complex 
military  problems  is  no  longer  effective  nor  sufficient 
(14:46-47) . 

Program  managers  gave  three  main  reasons  why  they  do 
not  regularly  use  quantitative  decision  support  techniques 
in  helping  them  discover  possible  solutions  to  their 
problems  (28:77,  40:38).  First,  they  expressed  reluctance 
to  use  quantitative  techniques  because  of  the  effort 
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involved  in  doing  the  calculations  manually  or  having  them 
done  by  a  mainframe  computer  expert.  The  managers  commented 
next  on  the  difficulty  in  learning  how  to  use  the  software. 
Third,  th-ey  disliked  the  decision  making  process  delay 
caused  by  the  time  spent  explaining  the  situation  to  a 
computer  expert  and  the  wait  for  the  necessary  analysis 
information  feedback  (48:69-70). 

Sped  f  ic  Problem 

Does  exposing  an  Air  Force  system  acquisition  midlevel 
manager  to  a  specific  computer-aided  quantitative  decision 
support  technique,  in  an  academic  environment.  Increase  the 
frequency  the  manager  uses  any  decision  support  technique  in 
actual  program  and  project  problems? 

Research  Qb j ectives 

A  quantitative  decision  support  technique,  decision 
tree  analysis,  is  taught  to  Air  Force  system  acquisition 
program  managers  in  the  Intermediate  Program  Management 
(SYS-400)  Professional  Continuing  Education  (PCE)  course  at 
Wrlght-Patterson  AFB  OH.  Before  this  research  began,  the 
classes  were  doing  the  decision  tree  calculations  without 
the  aid  of  a  computer.  This  study  focused  on  determining 
whether  modifying  the  SYS-400  curriculum  to  teach  the  use  of 
PC-based  decision  tree  algorithms  to  program  managers  would 
increase  the  amount  program  managers  used  any  quantitative 
technique  in  addressing  actual  program  and  project  problems. 
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The  following  was  used  as  a  guide  for  this  research; 

1.  How  often  do  managers  use  quantitative  decision  support 
techniques,  with  or  without  the  aid  of  a  computer? 

2.  What  tools  do  managers  need  to  problem  solve?  What  do 
they  want  to  use  for  help? 

3.  Does  a  three  month  after-course  survey  of  the  treatment 
groups  provide  reliable  results  that  can  be  compared  to  the 
three  month  after-course  surveys  of  the  control  groups? 

4.  What  demographic  characteristics  effect  the  frequency 
of  quantitative  decision  support  technique  use? 

3.  The  Defense  System  Management  College  has  identified 
several  decision  support  techniques,  listed  below,  that  are 
useful  to  program  managers.  How  often  do  program  managers 
use  the  following  decision  support  techniques? 

a.  Life  cycle  cost  analysis  (13:17-8;  43:44-43) 

b.  Trade-off  analysis  (13:8-1;  43:19) 

c.  Statistical  analysis 

d.  Linear  programming  (13:15-2) 

e.  Ranking 

f.  Risk  and  uncertainty  analysis  (13:15-1;  11:1-2;  43:19) 

g.  Cost  driver  sensitivity  analysis  (13:17-6;  43:41) 

h.  Reliability  analysts  (13:19-2) 

1.  Utility  analysis  (13:8-8) 

j.  Simulation  (13:6-6) 

k.  Program  network  schedules  (12:4-9;  43:18) 

l.  Schedule  management  (such  as  program  status  and 
identifying  critical  activities)  (43:35) 

m.  Conducting  schedule  risk  assessments  (11:11-1; 

43:37,45) 

n.  Estimating  program  milestones  (13:9-1;  15:3-1;  12:3-1) 

o.  Decision  tree  analysis  (ll:IV-6) 

p.  Project  Evaluation  and  Review  Technique  (PERT)  and 

Critical  Path  Method  (CPM)  analyses  (ll:IV-2;  15:8-1; 

12:4-1) 

q.  Developing  alternate  plans  to  meet  objectives  at 
the  beginning  of  a  program  (43:40) 

r.  Cost/Schedule  Control  System  Criteria  (C/SCSC)  (13:9-1) 

s.  Financial  methods  (43:42-44) 


Scope  of  this  Study 

This  study  investigated  how  often  Air  Force  system 
acquisition  program  managers  use  quantitative  decision 
support  techniques.  Does  teaching  a  PC-aided  decision 
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support  technique  (specifically,  decision  tree  analysis) 
increase  how  often  a  program  manager  use  any  quantitative 
decision  support  technique?  A  research  experiment  was 
developed  to  investigate  this  question.  The  study  was 
limited  to  Air  Force  midlevel  program  managers  who  attended 
the  Intermediate  Program  Management  PCE  course  at 
Wrlght-Patterson  AFB  OH.,  from  October  1988  to  March  1989. 
The  variable  studied  was  “How  often  managers  use  these 
techniques  to  aid  solving  program  problems.* 

An  after-course  survey  was  used  to  gather  informai,ion 
on  the  use  of  several  different  quantitative  techniques  to 
measure  the  effect  of  the  treatment,  classroom  instruction. 
Demographic  data  was  collected  to  determine  if  they  effected 
the  frequency  of  use.  The  demographic  effects  might 
disguise  the  treatment  effects. 

This  study  does  not  try  to  discover  if  other  teaching 
techniques  or  other  situations  will  alter  the  frequency  of 
quantitative  technique  use.  Determining  and  validating  a 
set  of  demographic  predictors  for  the  program  managers  who 
are  most  receptive  to  using  quantitative  decision  support 
techniques  is  left  for  future  research. 

Def ini tlons 

Management  Science . 

A  scientific  approach  to  decision  making,  which 
seeks  to  determine  how  best  to  design  and  operate  a 
system,  usually  under  conditions  requiring  the 
allocation  of  scarce  resources.  (52:2) 
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System  Program  Of  f ioe  (SPO)  . 


The  SPO  is  the  basic  organizational  structure  in 
Air  Force  Systems  Command  responsible  for  purchasing 
new  weapons  and  support  systems  for  the  Air  Force.  If 
the  SPO  is  responsible  for  only  one  system,  it  la 
called  a  single  system  SPO.  If  the  SPO  la  responsible 
for  more  than  one  system,  it  la  called  a  basket  SPO. 

An  example  of  a  single  system  SPO  la  the  B-1  SPO.  An 
example  of  a  basket  SPO  la  the  Airlift  and  Trainer  SPO. 
(28:3) 

Program/Pro j ect  Manager .  A  project  manager  is 
responsible  for  all  functions  needed  for  a  change  to  the 
organization  (2:61) .  The  change  is  the  project  and  could  be 
as  diverse  as  the  purchase  of  a  new  product  line,  the 
implementation  of  a  new  organizational  structure,  the  design 
and/or  implementation  of  a  new  organizational  strategic 
plan,  etc.  (2:61-63).  The  difference  between  a  program 
manager  and  a  project  manager  is  one  of  scale.  A  program 
manager  Is  responsible  for  more  than  one  project.  This 
study  uses  the  terms  ‘program  manager’  and  ‘project  manager' 
to  mean  the  same  thing. 

Decision  Support  System  (DSS)  . 

A  decision  support  system  is  a  man-machine  couple 
that  facilitates  incorporation  of  experience  and 
instinct  in  decision  making.  Using  electronic  data 
recall,  manipulation  and  graphic  aisplay  to  augment 
managerial  Judgement,  decision-makers  can  review  and 
edit  their  choices  selectively  before  implementing  an 
Irrevocable  decision.  (23:24) 


Summary 

This  chapter  has  provided  the  general  background  on  the 
initiation  of  this  thesis.  There  are  numerous  demands  for 
the  program  manager's  time  and  decisions.  However, 
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sophisticated  decision  support  tools  can  both  save  the 
program  manager’s  time  and  Improve  the  decision  process. 
Program  managers  have.  In  general,  been  reluctant  to  tap 
Into  this  resource.  The  hypothesis  la  that  11  program  and 
project  managers  are  taught  to  use  a  computer  to  perform  the 
difficult  computations  of  a  quantitative  decision  support 
technique,  they  are  more  likely  to  apnly  that  technique,  and 
possibly  others,  to  actual  problems  that  occur  In  their 
respective  Jobs. 

Chapter  II  gives  a  more  detailed  background  on  the  role 
of  a  project  manager,  characteristics  of  their  work.  It 
also  describes  how  computers  In  the  program  office  and 
quantitative  decision  support  techniques  can  be  beneficial 
within  that  environment.  Initially,  the  roles  of  system 
acquisition  program  manager  are  discussed,  focusing  on  two 
approaches  to  program  management,  problem  solving  patterns 
and  types  of  decisional  views.  Next,  the  characteristics  of 
the  managers  work  and  the  system  acquisition  program 
managers  environment  are  described.  This  Is  followed  by  an 
examination  of  program  management  aids,  that  Is,  how  the 
manager  can  get  more  time  to  plan  and  analyze  solutions  to 
problems  with  computerized,  standard  output.  An  examination 
of  how  a  decision  support  system  must  work  within  the 
managers’  environment,  problem  solving  patterns  and 
decisional  views  to  be  beneficial  Is  discussed.  The  chapter 
concludes  with  a  description  of  the  Management  Science 
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decision  support  tools  that  are  available  to  system 
acquisition  program  managers. 

Chapter  III  describes  the  methodology  of  the  experiment 
and  the  statistical  techniques  used  to  collect  Information 
from  the  data.  Chapter  IV  presents  the  Information  gathered 
from  the  experiment,  while  Chapter  V  details  the  conclualona 
of  this  research  and  some  recommendations  for  future 
research . 
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Literature  Review 


II . 


Introduction 

Thla  chapter  provldea  background  for  the  enaulng 
research.  Before  Investigating  the  question  "Why  do  Air 
Force  acquisition  system  program  managers  rarely  use 
quantitative  decision  support  techniques?",  there  must  be  an 
understanding  of  what  such  managers  do  and  what  type  of 
environment  they  work  in.  How  a  manager  solves  program 
problems  and  how  the  computer  can  help  the  manager  must  also 
be  understood. 

Initially,  the  roles  of  system  acquisition  program 
manager  are  discussed,  focusing  on  two  approaches  to  program 
management,  problem-solving  patterns  and  types  of  decisional 
views.  Next,  the  characteristics  of  the  managers  work  and 
the  system  acquisition  program  managers  environment  are 
described.  Thla  is  followed  by  an  examination  of  program 
management  aids,  that  is,  how  the  manager  can  get  more  time 
to  plan  and  analyze  solutions  to  problems  with  computerized, 
standard  output.  An  examination  of  how  a  decision  support 
system  must  work  within  the  managers’  environment,  problem 
solving  patterns  and  decisional  views  to  be  beneficial  la 
discussed.  The  chapter  concludes  with  a  description  of  the 
Management  Science  decision  support  tools  that  are  available 
to  system  acquisition  program  managers. 
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Syst'^’n  Aogulal tlon  Program  Manager  Rolea 


Mlntzberg  deacribea  the  three  managerial  rolea  aa 

Interpersonal,  informational  and  deciaional  (34:58).  The 

Project  Management  Inatitute,  publlaher  of  the  periodical 

Program  Manager .  further  deacribea  the  rolea  with  a  Hat  of 

several  project  manager  functions; 

planning  and  scheduling,  performance  analysis,  progress 
reporting,  maintaining  cl lent/conaul tant  relations, 
project  trend  analysis,  coat  trends  analysis,  logistics 
management,  coat  control,  organization  and  manpower 
planning,  maintaining  the  technical/bualnesa  interface, 
contract  administration,  controlling  materials  and 
manpower,  estimating,  and  procedure  writing  and 
adm<  n  1  ?» trnt  ion  .  (!:13) 

The  decisional  role  is  separated  into  four  sub-roles,  the 
entrepreneur,  the  disturbance  handler,  the  resource 
allocator  and  the  negotiator  (34:58).  How  should  a  systems 
acquisition  program  manager  approach  decision  nvaklng  in 
accomplishing  these  rolea  and  functions?  The  next  section 
examines  this  question. 

Approaches  to  Program  Management .  Pox  and  Field 

describe  two  types  of  DoD  project  manager’s  views  on 

decision  making,  the  liaison  manager  view  and  the  active 

manager  view  (20:302) .  They  describe  the  liaison  manager  aa 

needing  little  project  manager  experience  and  coat  control 

experience  and  la  therefore, 

limited  to  promoting  the  program,  preparing  progress 
reports,  and  performing  technical  liaison.  The  liaison 
type  of  manager  relies  on  the  contractor  for  coat 
control.  (20:302) 
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The  active  manager  view 

la  baaed  on  the  belief  that  the  competitive  forcea  of 
the  market  place  do  not  produce  the  dealred  coat, 
achedule,  and  technical  performance  on  large  defenae 
ayatema.  (20:302) 

The  active  manager  muat  be  actively  Involved  in  and  familiar 
with  all  phaaes  and  proceaaea  of  the  project  (20:303) . 

Fox  and  Field  contend  that,  of  the  two  viewa,  the 
active  manager  la  the  preferred  management  approach.  They 
‘conalder  the  defenae  acqulaltlon  proceaa  too  complex,  too 
coatly,  and  too  Important  to  national  aecurlty  to  aerve  as 
an  alternate  aaalgnment*  aa  a  llalaon  manager  would 
(20:304).  The  llalaon  manager  la  primarily  concerned  with 
the  1 nterperaonal  and  informational  typea  of  rolea  and 
functlona.  The  active  manager  la  equally  concerned  with  all 
the  managerial  rolea,  including  the  decisional  role. 

There  are  constant  demands  on  program  managers  for 
their  time  and  for  their  decisions.  Program  managers  must 
be  able  to  coordinate  and  Integrate  the  actions  and 
resources  of  several  different  organizations  aa  well  aa 
their  own  system  program  office  (SPO)  (1*:46) .  Active 
managers  must  decide  how  to  minimize  the  conflicts  between 
the  program’s  performance,  schedule  and  coat.  Pressure  la 
exerted  for  the  project  to  meet  supportabl 1 1 ty ,  reliability 
and  many  various  other  goals. 

In  today’s  highly  complex  and  volatile  environment,  the 
program  manager  for  Department  of  Defense  (DoD)  weapon 
system  acqulaltlons  is  faced  with  having  to  make  intelligent 
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declslona  that  will  effect  the  United  States’  national 
interests  (2*1:165).  The  national  interests  are  important  to 
the  active  manager  when  considering  the  alternatives  in 
solving  problems  in  the  program  office. 

Problem  Solving  Patterns.  How  does  the  active  manager 
solve  problems  in  their  programs?  Problem  solving  is  the 
primary  function  of  managers  (2:2-4;  45:32).  The  complexity 
and  the  difficulty  of  solving  the  problem  is  what  motivates 
the  manager  to  think  about  and  develop  alternative  solutions 
(47:25) .  Stavenjord  lists  four  approaches  to  applying  that 
process.  One  of  these  four  patterns  are  generally  followed: 

1.  Crisis  management:  selecting  the  first  solution  that  is 
presented . 

2.  Cookbook  problem  solving:  using  a  ’laundry  list'  to 
match  the  problem  with  a  solution. 

3.  Reflective  thinking:  decision  based  on  analysis  of  the 
3 1 tuat i on . 

4.  ’Habitually  reflective  thinking:  automatic  reflective 
thinking,  always  step  back,  look  at  issue,  develop 
alternatives’,  and  arrive  at  a  decision  that  makes  sense  as 
the  best  solution  (47:25-26). 

The  manager  should  try  to  solve  problems  by  habitually 
reflective  thiniclng.  This  takes  planning,  data  and 
knowledge  to  transform  that  data  into  meaningful 
information.  Any  tool  that  is  made  available  to  managers 
must  enable  them  to  progress  to  constant,  habitually 
reflective  thinking.  But  what  does  a  program  manager  use  as 
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the  criteria  to  select  between  the  alternatives  discovered 
through  habitually  reflective  thinking? 

Decisional  Views .  There  are  several  reasons  a  project 
manager  chooses  a  specif Ic  solution.  Keen  and  Morton 
suggest  that  there  are  five  views  of  selecting  a  decision: 

1.  The  economic  rational  concept:  This  la  the 
classical  normative  theory  of  decision  making,  in  which 
decision  makers  are  all-knowing  and  able  to  evaluate 
all  alternatives.  They  are  dissatisfied  with  any 
solution  but  the  beat. 

2.  The  satisficing,  process-oriented  view:  This 
considers  the  decision  makers  to  be  intendedly  rational 
although  cognitive  limits  lead  to  a  bounded 
rationality;  thus  the  goal  of  any  decision  maker  is  to 
get  a  good  enough  answer,  not  the  beat  possible  one. 
This  point  of  view  stresses  the  process  of  decision 
making  and  not  Just  its  outputs;  it  emphasizes  the 
relatively  limited  analysis  and  search  moat  managers 
will  make  and  their  reliance  on  heuristics. 

3.  The  organizational  procedures  view:  This  focuses 
on  the  interrelations  among  components  of  the 
organization.  It  highlights  organizational  structure, 
mechanisms  for  communication  and  coordination,  and  the 
standard  operating  procedures  by  which  decision  making 
la  systemlzed  and  often  slmplifled. 

4-  The  political  view:  This  regards  the  participants 

in  the  decision  process  as  actors  with  parts  to  play. 
They  have  strong  individual  preferences  and  vested 
interests  and  form  coalitions  of  organizational 
subgroups  Decisions  are  frequently  dominated  by 
bargaining  and  conflict,  with  the  result  that  only 
small  deviations  from  the  status  quo  are  normally 
possible.  Major  Innovations  are  (quite  reasonably) 
resisted  by  those  who  position,  interests,  or  simply 
Job  satisfaction  will  be  affected 

5.  The  individual  differences  approach:  This  view 

argues  that  an  individual’s  personality  and  style 
strongly  determine  his  or  her  choices  and  behavior. 
Personal  'rationality*  la  subjective  and  behavior  la 
very  much  determined  by  the  manT>*»T«  in  which  an 
individual  processes  information.  (27:80) 
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Each  of  these  viewpoints  is  a  valid  selection  criteria  in 
different  problem  solving  situations.  A  decision  support 
system  might  be  Informative  for  one  viewpoint,  but 
Irrelevant  for  another. 

Selection  of  a  decisional  view  might  be  influenced  by 
the  managers’  background,  responsibility  areas  of  decisions, 
personal  approach  to  program  management ,  the  external 
pressures  of  the  Job,  or  various  other  factors.  In  what 
kind  of  environment  do  they  perform  their  managerial  roles? 
Do  active  managers  have  time  for  habitually  reflective 
thinking?  This  la  examined  next. 


Program  Management  Env ironment 

A  Massachusetts  Institute  of  Technology  researcher, 
working  with  the  DoD ,  documented  some  difficulties  of  a  DoD 
decision  maker. 

The  complexity  of  the  battlefield  imposes  severe 
constraints  on  the  human  beings  who  participate  in  the 
operations  The  decision-making  and  information 
processing  organizations  designed  to  execute  thes^ 
tasks  exist  in  hostile  environments  where  the  tempo  is 
fast  and  the  data  to  be  gathered  and  analyzed  are 

numerous.  Therefore,  _ ,  the  distribution  of 

processing  between  the  hardware  and  human  components  is 
a  necessity  in  order  to  facilitate  the  task  carried  out 
by  each  organization  member.  Their  activities  are 
synergistic  and  coordination  of  the  decision-making 
processes  must  be  achieved  in  order  to  Improve  the 
effectiveness  of  the  organization.  It  follows  that 
structures  which  allow  the  coordination  of  the 
different  activities  neceaaary  to  fulfill  the  mission 
must  be  designed.  However,  decision  aids  also  increase 
the  possible  alternatives  among  which  to  choose  in 
order  to  process  information,  and  in  so  doing,  modify 
the  nature  of  the  dec  1 s 1  on- maker s ’  activities.  (22-  19) 
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The  hectic  pace  of  a  battlefield  is  aimilarly 
characterized  in  the  activity  in  the  program  office.  Juat 
as  battlefield  commanders  have  numerous  staff  personnel 
providing  them  Information,  the  SPO  program  manager  has  a 
staff  to  provide  information.  There  are  five  different 
divisions  in  the  SPO: 

1.  The  program  control  ...  division  directs  overall 
planning,  programming,  collection  and  analysis  of  cost 
and  schedule  data,  performance  reporting  to  higher 
levels  in  DoD,  and  financial  management. 

2.  The  configuration  management  division  establishes 
and  implements  policies  and  procedures  for 
configuration  management:  system  equipment  and  facility 
identification  specification,  engineering  change 
control,  and  performance  reports  on  all  such 
activities . 

3.  The  procurement/contracting  and  production  division 
manages  all  procurement  and  production  activities  and 
supervises  the  planning  and  execution  of  all  contracts 
for  research  studies,  engineering  development,  tests, 
and  production. 

4.  The  systems  engineering  division  manages  the  total 
systems  engineering  function,  including  the  integration 
of  engineering  systems  and  subsystems.  This  division 
is  also  responsible  ^or  the  quality  of  the  technical 
performance  of  the  weapon  system. 

5.  The  product  assurance  (or  test  and  deployment) 
division  plans  and  coordinates  the  test  programs  for 
the  weapons  system.  (20:157) 

How  managers  use  their  time  has  been  meticulously 
studied.  Mlntzberg  conaucted  a  study  of  five  chief 
executives  who  were  directly  observed  for  five  weeks 
(33:25,32)  and  described  the  managers’  day  as  disjointed, 
characterized  by  numerous  short  activities  and  constant 
interruptions  (33:25-26,  29:19).  Each  of  the  managers' 
activities  averaged  less  than  twenty  minutes 
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except  scheduled  meetings  (33:33).  "Hall  of  the  observed 
activities  were  completed  in  less  than  nine  minutes,  and 
only  one-tenth  took  more  than  an  hour'  (33:33). 

Managers  structure  the  organization  to  provide  the  most 
current  information  as  fast  as  possible.  That  is  the  reason 
for  the  constant  interruptions  and  disjointed  activities. 

The  manager  'demonstrates  a  strong  thirst  for  current 
information  and  ...  tends  to  do  little  with  routine  reports 
that  his  organization  provides  for  him"  (33:36).  Luthans , 
the  president  of  the  Academy  of  Management,  presented 
similar  information  in  a  report  for  the  Academy  of 
Management’s  (31:8). 

Work  Characteristics .  Mlntzberg  determined  thirteen 
characteristics  of  a  manager  at  work.  The  manager: 

1.  la  'compelled  to  perform  a  quantity  of  work....  Breaks 
are  rare'  (33:51). 

2.  is  'characterized  by  brevity,  variety  and 
fragmentation'  (33:51). 

3.  "prefers  brevity  and  interruptions'  (33:51). 

4.  "gravitates  to  the  more  active  elements  of  his  work' 
(33:51).  Very  current  information  is  highly  prlzeds. 

5.  'favors  the  verbal  media,  spending  most  of  his  time  in 
verbal  contact'  (33:51). 

6.  gives  mall  "cursory  treatment' ,  ...  writes  "much  less 

mall  than  he  receives'  (33:52). 

7.  uses  'telephone  and  unscheduled  meetings  . . .  for  brief 
contacts  . . .  and  when  information  or  requests  must  be 
transmitted  quickly'  (33:52). 

8.  schedules  meetings  that  'consumes  more  . . .  time  than 
any  other  medium’  (33:52) . 

9.  spends  'little  time  ...  in  open-ended  touring"  (33:52) . 
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10.  has  "external  contacts  [that]  generally  consume 
one-third  to  one-half  of  the  manager’s  contact  time" 

(33:52) . 

11.  allows  subordinates  to  "generally  consume  one-third  to 
one-half  of  the  manager's  contact  time"  (33:53). 

12.  spends  "relatively  little  of  his  time  with  his 
superior  -  generally  on  the  order  of  ten  percent"  (33:53). 

13.  "is  responsible  for  many  initial  commitments,  which 
lock  him  into  a  set  of  ongoing  activities"  (33:53) . 

The  program  manager  is  constantly  Inundated  with  staff 
information  and  demands  for  decisions  based  on  that 
information.  The  manager  must  balance  the  conflicting 
requirements  of  the  program’s  schedule,  cost,  performance 
maintainability  and  numerous  other  criteria.  Trade-offs  and 
compromises  are  constantly  evaluated.  The  program  manager 
must  accomplish  all  of  this  with  a  limited  number  of  people 
and  a  limited  amount  of  time. 

Most  of  a  manager’s  time  is  spent  in  short  bursts  of 
activity  on  numerous  different  activities.  A  manager  cannot 
devote  an  extended  period  of  time  on  any  single  problem  or 
activity.  It  appears  that  managers  would  placr  significant 
value  on  "time  savers"  or  "time  multipliers",  which  allow 
them  to  accomplish  several  activities  at  the  same  time.  How 
a  manager  can  accomplish  this  is  examined  next. 

Program  Management  Aids 

Two  "time  savers"  available  to  the  program  managers  are 
computerized  standard  output  and  computerized  decision 
support  techniques,  decision  support  systems.  Implemented 
effectively,  these  could  provide  the  managers  with  time  and 
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Information  needs  for  habitually  reflective  thinking.  This 
facilitates  planning  to  minimize  or  prevent  future  program 
problems  and  developing  alternative  solutions  to  current 
problems.  The  benefits  and  disadvantages  of  each  technique 
are  explored  next. 

S tandard 1 zed  Output .  There  are  numerous  advantages  for 
using  a  personal  computer  in  a  program  manager’s  office 
(30:33) .  Just  as  a  hammer  Is  given  to  a  carpenter  to  help 
accomplish  a  task,  microcomputers  have  been  provided  to 
managers  to  help  them  do  their  work  (5:2) .  John  T.  LeSueur , 
an  Air  Force  system  acquisition  program  manager,  listed  four 
major  areas  with  an  example  for  each:  word  processing  (spell 
checking) ,  electronic  mall  (information/  data  exchange) , 
spreadsheets  (fast  reaction  to  changes  and  showing  various 
viewpoints)  and  program  management  tools  (PERT/Qantt  charts 
and  tracking  tasks  to  be  accomplished)  (30:33-34). 

Another  program  manager,  Michael  F.  Turner,  listed 
several  other  benefits:  reduction  of  manual  tasks,  decrease 
In  the  number  and  the  length  of  interruptions,  the 
availability  of  a  centralized  Information  storage  area,  the 
decrease  in  document  turn  around  time,  the  speed  of 
distributing  documents  and  the  ease  of  making  revisions, 
retrieving  information,  automatic  filing,  scheduling 
meetings  and  delegation  of  duties  (49:14). 

Advantages .  The  literature  noted  there  are  two 
major  benefits  from  using  computers  for  standardized  output 
In  the  program  manager’s  office.  First,  the  ability  to 
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generate  similar  reports  quickly  and  accurately  can  save 
time  for  the  manager  (49:13).  A  faster  learning  process  la 
the  second  advantage  of  computerizing  standardized  output. 
Specific  guidelines  are  normally  established  for  the  format 
of  reports  (53:72;  9:43).  The  same  margins,  ratios, 
displays  of  Information  (tables  and  graphs),  and  the  use  of 
preprinted  forma  are  difficult  to  learn  Initially  and 
detract  from  the  manager's  more  Important  Job  of  managing. 
Computerized  output  speeds  up  the  learning  process  for  the 
new  manager  (53:73). 

Disadvantages .  There  are  two  main  disadvantages 
of  using  computers  for  standardized  output  In  the  program 
manager’s  office.  It  is  difficult  to  implement  any  changes 
to  the  ’old  habits*  and  it  la  difficult  to  learn  the  new 
working  procedures  by  using  the  new  computer  (49:13).  Most 
of  the  planning  time  Is  spent  on  which  computer  system  to 
acquire  and  how  to  install  them.  Little,  If  any,  time  is 
used  to  plan  new  work  techniques,  learning  requirements,  and 
the  use  of  the  new  computer  to  aid  project  success  (49:17). 
Initially,  there  may  be  a  high  error  rate  In  report  output 
or  even  an  Increase  in  report  accomplishment  time  while  the 
worker  learns  how  to  use  the  new  computer  (44:707) . 

Using  a  computer  to  do  repetitive  tasks  allows  more 
time  for  the  manager  to  concentrate  on  problem  solving.  The 
disadvantages  can  be  minimized  with  proper  planning  and 
integration  with  user  needs  and  desires.  Training  before 
computer  installation  can  decrease  the  disadvantages.  Is 
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the  computer  an  aid  in  solving  problems?  The  next  ‘ time 
saver’s',  computerized  decision  support  systems,  advantages 
and  disadvantages  are  discussed  next. 

Problem  Solving  Tool.  Only  nine  percent  of  the  program 
managers  stated  they  often  used  a  quantitative  decision 
support  technique,  statistical  analysis  (28:77).  Why  don't 
these  managers  use  this  tool?  Is  it  because  it  wasn’t 
computerized? 

Tools  used  by  the  manager  need  to  complement  Keen  and 
Morton’s  view  of  managerial  of  decision  making;  economic 
rational,  satisficing,  organizational  procedures,  political 
views  and  individual  differences.  A  tool  should  fit  into 
the  managerial  environment,  support  short  activities  and 
numerous  Interruptions,  and  provide  current  Information. 
Mintzberg  described  a  three  step  decision  making  process. 
First,  identification;  the  recognition  and  diagnosis  of  the 
need  for  a  decision.  Second,  development  of  solutions;  the 
search  and  design  of  alternative  solutions.  Third  la  the 
selection,  screening,  evaluation  and  authorization  of  the 
selected  solution  (34:58).  A  decision  support  technique 
must  support  all  steps  of  this  process. 

The  active  manager  should  obtain  as  much  information 
from  several  problem  perspectives  before  a  crisis  exists, 
which  could  lead  to  a  more  rational,  informed  decision  and 
habitual  reflective  thinking.  A  tool  to  assist  in  achieving 
that  level  of  thinking  is  the  computerized  DSS .  The 
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advantages  and  disadvantages  of  a  computerized  DSS  are 
discussed  next. 

Advantages .  A  decision  support  tool  available  to 
the  manager  Is  the  computerized  decision  support  system 
(DSS)  (23:24;  9:43).  The  DSS  can  guide  the  manager  through 
several  alternatives  to  specific  problems  or  simulation  of 
problems  (9:43;  19:5;  23:24;  30 : 35 ; 37 : 27) .  Being  exposed  to 

several  options,  the  manager  can  develop  better  solutions  to 
a  problem  than  If  the  manager  used  crisis  management  or 
textbook  solutions  that  do  not  entirely  match  the  real 
problem  parameters  (47:25).  These  alternatives  guide  the 
manager  to  leaning  different  techniques  for  problem  solving 
(25:1646).  These  benefits  lead  to  habitually  reflective 
thinking . 

A  decision  support  system  can  be  used  to  track  the 
Implementation  effectiveness  of  the  decision.  It  provides 
Information  that  helps  the  decision  maker  choose  which 
variables  of  the  problem  are  most  Important  and  should  be 
monitored.  The  decision  support  system  can  help  the 
manager’s  office  'execute  their  project  In  a  more  effective 
and  efficient  manner*  (9:43). 

The  "What  if?'  capability  of  a  decision  support  system 
allows  the  manager  to  plan  ahead  for  future  problems  and 
possibly  prevent,  or  at  least  minimize,  the  effect  of  some 
problems  (18:2,  23:24).  Additionally,  the  display  of 
alternative  solutions  to  the  same  problem  gives  insights  to 
the  manager  on  considering  different  approaches  from 
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different  decision  making  methods  (18:3,  9:43;  19:10).  This 

advantage  facilitates  planning  and  problem  minimization 
before  the  problem  develops  Into  a  crisis. 

Computerized  decision  tools  are  desired  by  the  majority 
of  project  managers  (28:81-84).  The  computerized  decision 
support  system  helps  the  project  manager  to  react  faster  and 
more  intelligently  to  the  pressures  from  a  superior’s 
request  for  additional  Information  and  the  constant  program 
trade-offs  required  between  cost,  performance,  and  schedule 
of  the  project. 

Computerized  decision  support  systems  aid  the  manager 
In  making  better  decislona,  but  will  not  replace  good 
Judgement  (9:43) .  They  do  not  give  the  decision  maker  the 
’right  answer*;  only  the  decision  maker  can  decide  what  is 
correct  for  their  specific  situation  (18:3).  Instead, 
decision  support  systems  give  the  manager  a  rational  set  of 
alternatives  and  teach  the  manager  different  methods  of 
problem  solving  (18:3,  9:43).  They  allow  the  manager  to 
focus  on  the  most  significant  problem  and  gives  the  manager 
the  capability  to  test  a  solution  (simulation)  before  the 
solution  Is  Irrevocably  Implemented  (23:24;  47:25). 

Decision  support  systems  allow  the  manager  to  plan 
ahead  for  possible  future  problems  by  asking  ‘What  if?’  type 
questions  (23:26) ,  and  information  to  prevent  a  problem  from 
occurring  (9:43) .  Thirty  to  fifty  percent  of  the  SPO 
program  managers’  time  is  spent  promoting  and  defending 
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their  programs  to  upper  management  (20:302).  They  must  be 
able  to  justify  their  decisions  with  rational  information. 

The  computer  allows  the  manager  to  delegate  to  the 
office  staff  the  information  gathering,  suggestion  of 
alternatives  and  implementation  of  solutions  (23:26).  This 
delegation  helps  the  manager  to  avoid  being  what  General 
Skantze,  former  commander  of  USAF  Systems  Command,  calls  a 
'good  news  program  manager'  (46:20).  The  better  Informed 
the  program  manager  la,  the  more  successful  the  manager 
(46:20) . 

Disadvantages .  There  are  numerous  pitfalls  in 
using  a  computerized  decision  support  system.  The  problems 
faced  by  a  specific  program  manager  are  unique  (51:501). 
Program  managers  prefer  to  set  specific  boundaries  and 
relationships  for  their  individual  project  and  think  that 
the  project  will  remain  static  (47:24).  However,  the 
manager’s  program  is  always  changing.  Writing  a  computer 
program  that  will  solve  some  future  problem  in  a  program 
that  is  dynamic  is  difficult  (53:72) . 

Various  computer  programs  are  written  to  solve  some 
short  term  problem  or  are  very  narrowly  delimited.  This 
often  causes  program  managers  to  increase  the  number  of 
items  they  try  to  manage  without  gaining  any  real  control  of 
the  problem  that  needs  solving  (4:36). 

The  manager  may  try  to  influence  every  minute  detail, 
even  if  that  detail  does  not  contribute  significantly  to  the 
accomplishment  of  program  goals  (4:36).  Some  decision 
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support  systems  rearrange  the  data,  but  not  In  a  very 
meaningful  pattern.  (5'^;72).  The  manager  may  be  creating 
Information  reports  Just  for  the  reports  sake  and  not  for 
the  Information  (4:36) .  This  can  lead  to  short  term  goal 
achievement  that  could  be  counterproductive  to  the  long  term 
goals  of  the  program  office  (3:39). 

The  common  user  of  computer  programs,  like  a  decision 
support  system.  Is  not  a  computer  specialist  (8:2441,  a 
significant  point  since  some  computer  programs  are  difficult 
to  learn  and  to  use  (25:1646,  53:73).  This  fact  normally 
causes  the  computer  programs  to  be  only  ’expensive 
novelties’  that  sit  or  the  shelf  gathering  dust  (53:72). 

Some  programs  only  accomplish  part  of  the  user’s  goals  and 
leave  the  user  frustrated  (32:340).  Some  decision  support 
system  models  are  too  simple  or  are  designed  to  make  the 
same  decision  mistakes  the  programmer/exper t  might  make 
(9:43,  51:501,  53:73) . 

Incompatibility  of  computer  hardware  systems  Is  another 
problem  (9:43).  The  Information  may  be  unavailable  to  other 
computer  systems  or  programs.  More  work  might  be  needed  to 
update  several  computer  systems  than  If  a  manual  system  was 
used  (41:29) .  The  tasks  assigned  to  subordinates  of  the 
manager  may  have  to  be  redefined  and  the  Job  may  become 
boring  and  repetitious  (21:356). 

Program  Management  Aids  Summary .  In  order  for  a 
computerized  decision  support  system  (DSS)  to  help  the 
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manager.  It  should  complement  the  manager’s  decisional 
views.  A  DSS  must  fit  into  the  manager’s  hectic  schedule. 

It  .Thould  provide  very  current  Information,  take  little  time 
to  use  and  provide  uncertain,  ad  hoc,  external  information 
(33 : 195-196) . 

The  goal  of  a  decision  support  system  is  to  take  the 
necessary  data  and  transform  it  into  a  form  that  is 
meaningful  to  the  manager.  The  manager  is  sometimes 
overwhelmed  with  reams  of  printouts  that  are  not  useful  and 
often  confusing.  If  the  form  isn’t  informative,  the 
decision  support  system  will  not  be  used. 

Project  managers  have  expressed  a  need  for  certain 
decision  support  techniques  to  be  computerized  (28:di-85). 
Computers  should  be  used  to  assemble  data,  synthesize  it 
into  information  and  then  correlate  and  distribute  the 
information . 

Some  of  the  available  Management  Science  decision 
support  techniques  are  examined  next. 

Management  Tools 

The  literature  notes  that  the  program  manager  has 
several  good  Management  Science  tools  available  to  aid  in 
the  decision  making  process.  The  Defense  Systems  Management 
College  (DSMC) ,  a  government  college  created  explicitly  to 
train  program  directors  and  managers,  provides  training  in 
using  these  quantitative  decision  support  techniques.  The 
DSMC  Systems  Engineering  Management  Guide  defines  several  of 
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the  tools.  All  of  these  tools  have  been  computerized. 
However,  The  tools  are  still  beneficial  without  a  computer. 

Life  Cycle  Cost  (LCC)  -  the  total  coat  to  the 
government  of  acquisition,  ownership  and  disposal  of  a 
system  over  its  entire  life  (13;17-8).  Life  Cycle  Coat 
Analysis  is  the  structured  study  of  LCC  estimates  and 
elements  to  Identify  life  cycle  cost  drivers,  total  cost  to 
the  government,  coat-risk  Items  and  cost-effective  changes 
(13: 17-8)  . 

Trade-o  f  f  Analya 1 s  -  *Trade-off  analysis  methodology 
provides  a  structured,  analytlca’  framework  for  [evaluating] 
a  set  of  alternative  concepts  or  designs’  (13:8-1)  . 

Risk  Managemen -  Risk  and  uncertainty  'management  la 
an  organized  means  of  Identifying  and  measuring  risk  and 
developing,  selecting  and  managing  options  for  handling 
these  risks’  (13;  15-1)  . 

Risk  Analysis  -  "determines  the  probability  of  events 

and  the  consequences  associated  with  their  occurrence’ 

(13:15-1).  Also,  risk  analysis  can  ’discover  the  cause, 

effects  and  magnitude  of  the  risk  perceived,  and  to  develop 

and  examine  alternative  options’  (13:15-1).  DSMC ’ a  ’Risk 

Assessment  Techniques'  defines  Risk  Assessment  as  a 

mathematical  analysis  of  the  probability  of  achieving 
or  not  achieving  acquisition  program  cost,  schedule  or 
performance  goals  (12;B-4). 
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PERT/CPM  (Program  Evaluation  and  Review  Technique/ 


Critical  Path  Method)  -  Plan,  achedule  and  control  a 
projects  time  aspects  and,  with  some  modification,  coat 
aspects  (36:298). 

A  graphical  portrayal  of  the  Interrelationships  among 
the  elements  of  a  project,  and  an  arithmetic  procedure 
which  Identifies  the  relative  Importance  of  each 
element  In  the  over-all  achedule.  (36:4) 

Decision  Tree  Anal ys 1 s  -  ‘A  graphical  representation  of 
a  decision  problem  used  to  determine  optimal  choices’ 
(52:587).  This  Includes  risk  aversion,  utility  theory,  and 
expected  value  of  information  theory  (52:601-602) . 

Schedule  Management  -  resolving  conflicts  between  a 
program’s  schedule,  cost  and  resou»'ces  (13:2-3)  . 

Reliability  -  'sustained  operational  performance  over 
time’  (16:1).  The  process  of  measuring  reliability  is  well 
described  in  HQ  AF ’ a  ’DSAF  R&M  2000  Process’ ,  Oct  88. 

Cone 1  us  1 ons 

This  review  has  focused  on  the  literature  discussing 
management  roles,  especially  problem  solving  patterns, 
decisional  views  and  the  characteristics  of  management  work. 
The  manager’s  day  is  characterized  by  numerous  interruptions 
and  short  bursts  of  activity.  The  utility  of  computerizing 
standard  output  and  problem  solving  In  the  program  manager’s 
office  was  discussed.  Computerization  has  many  benefits 
that  outweigh  the  possible  disadvantages.  Producing 
standardized  output  from  the  program  manager’s  office  can 
save  time  and  increase  productivity. 
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A  manager  can  make  a  better  decision  with  more 


information.  A  good  decision  support  system  makes 
information  available  quickly  and  easily.  Decision  support 
systems  help  managers  learn  different  techniques  for  problem 
solving.  However,  if  the  computer  programs  are  difficult  to 
learn  or  do  not  satisfy  the  office’s  goals,  many  of  those 
gains  are  lost.  Dangerous  computerized  decision  errors  must 
be  prevented  in  order  for  managers  to  accept  the  decision 
support  systems.  Computer  hardware  compatibility  is  also 
vital . 

The  literature  concludes  that  even  though  there  are 
many  tools  available  to  the  program  manager,  the  tools  are 
not  used  very  often.  Mintzberg  would  attribute  this  to  the 
lack  of  immediacy  of  the  information.  The  time  involved  In 
setting  up  the  new  system  may  be  another  detriment.  Before 
any  decision  support  system  is  implemented,  all  of  these 
factors  should  be  considered. 

Do  system  acquisition  program  managers  resist  using 
quantitative  decision  support  techniques  because  the 
techniques  aren’t  computerized?  The  hypothesis  Is  that  If 
program  and  project  managers  are  taught  to  use  a  computer  to 
perform  the  difficult  computations  of  a  quantitative 
decision  support  technique,  they  are  more  likely  to  apply 
that  technique,  and  possibly  others,  to  actual  problems  that 
occur  in  their  respective  Jobs.  A  research  experiment  was 
conducted  to  investigate  this  question.  Chapter  III 
describes  the  methodology  of  this  experiment.  Chapter  IV 
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presents  the  results  of  the  experiment.  Chapter  V 
the  conclusions  and  the  recommendations  for  future 


discusses 


research . 


Ill . 


Methodology 


Introduction 

Do  system  acquisition  program  managers  resist  using 
quantitative  decision  support  techniques  because  the 
techniques  aren’t  computerized?  A  research  experiment  was 
conducted  to  investigate  this  questions.  This  chapter 
describes  the  methodology  used  to  study  that  problem. 

System  acquisitions  program  managers  rarely  use 
quantitative  decision  support  techniques  to  aid  their 
decision  making.  The  hypothesis  is  that  if  program  and 
project  managers  are  taught  to  use  a  computer  to  perform  the 
difficult  computations  of  a  quantitative  decision  support 
technique,  they  are  more  likely  to  apply  that  technique,  and 
possibly  others,  to  actual  problems  that  occur  in  their 
respective  Jobs.  To  Lest  this  hypothesis  a  research 
experiment  was  developed.  The  experiment  Included  the 
treatment  of  teaching  decision  tree  analysis  with  a 
computerized  decision  support  system.  Approximately  three 
months  after  the  treatment,  a  after-course  survey  was 
administered  to  determine  the  results  of  the  treatment.  The 
survey  results  were  analyzed  to  address  the  hypothesis. 

This  chapter  describes  the  quas i -experiment ,  the  survey  and 
the  analysis  techniques  used.  The  results  of  the  analyses 
are  presented  in  chapter  IV. 


30 


Quasi -Experiment 

This  study  attempted  to  determine  11  altering  a  part  of 
a  Professional  Continuing  Education  (PCE)  course  for  Air 
Force  system  acquisition  managers  affected  the  frequency 
that  those  program  managers  use  quantitative  decision 
support  techniques.  This  was  accomplished  by  conducting  a 
posttest-only  design  with  nonequivalent  groups 

quasi-experiment  (7:98).  This  type  of  quasi-experiment  is  a 
research  experiment  conducted  on  a  control  group  and  a 
treatment  group  with  similar  average  member  demographics  not 
controlled  (7:34) . 

The  main  problem  with  this  type  of  teat  design  is  there 
la  no  pretest  to  compare  poattest  results  with  (7:149). 
Therefore,  differences  in  the  results  could  be  treatment 
orgroup  member  selection  (7:149).  This  problem  is  minimized 
by  careful  and  rigorous  examination  of  the  groups' 
demographics  and  the  other  variables  that  are  present 
(7:149).  The  control  group’s  poattest  results  were  compared 
to  the  treatment  group’s  postteat.  The  same  type  of 
comparisons  were  made  as  if  a  pretest  was  available. 

The  following  limitations  constrained  the  scope  of  this 
research : 

1.  A  quasi-experiment  was  chosen  because  a  large  random 
sample  of  project  managers  could  not  be  independently  and 
randomly  gathered  to  participate  in  and  to  complete  the 
after-course  survey  before  the  deadline  of  this  research. 
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2.  Interviews  with  the  selected  course  director  ruled  out 
using  a  control  group  and  a  treatment  group  from  the  same 
PCE  class.  The  concern  was  that  the  control  group  students 
might  have  become  dissatisfied  because  they  might  perceive 
that  they  were  made  to  do  more  work  than  the  treatment  group 
students . 

3.  A  pretest  was  not  used  because  one  of  the  control 
classes,  chosen  to  Insure  an  evaluation  of  a  large  enough 
number  of  students  for  this  effort,  had  already  completed 
the  course  before  this  research  started.  If  the  experiment 
was  delayed  to  administer  a  pretest,  there  would  not  have 
been  enough  students  taught  before  the  research  deadline. 

Student  Selection .  The  students  attending  the 
Intermediate  Program  Management  (SYS-400)  course  were 
selected  because  one  of  the  course  prerequisites  Is  that  the 
student’s  primary  Job  Is  related  to  Air  Force  system 
acquisition  and  program  management  (42).  Other  reasons  for 
using  the  SYS-400  students  were:  the  selection  of  the 
Individual  student  is  considered  random  when  comparing  one 
class  to  another;  course  prerequisites  and  student 
availability  are  the  main  criteria  for  selecting  a  student 
for  a  specific  class  (42);  and  the  course  already  presented 
a  quantitative  decision  support  technique  in  Its  curriculum. 

Four  classes  of  approximately  twenty-five  students  each 
were  selected  for  this  research,  the  first  two  for  the 
control  group  and  the  second  two  for  the  treatment  group, 
resulting  In  about  fifty  program  managers  In  each  group. 
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This  number  of  students  was  large  enough  to  allow  for  the 
typical,  sixty  percent,  survey  return  rate,  while  allowing 
for  a  statistically  large  enough  sample,  thirty,  of  the 
population  of  interest  to  conduct  the  appropriate 
statistical  analyses.  Only  those  students  that 
satisfactorily  completed  the  entire  course  of  instruction 
were  sent  surveys. 

Two  techniques  were  used  to  minimize  variable  effects 
other  than  the  treatment  variable.  First,  external 
environmental  affects  were  minimized.  Then  the  instruction 
variability  was  minimized.  External  environmental  effects 
are  changes  in  the  experiences,  societial  and  personal, of 
the  group  members  that  aren’t  controlled  or  measured  by  the 
experiment  (26:326).  An  example  of  an  external 
environmental  effect  would  be  a  change  in  the  management 
philosophy  of  the  organization  or  society.  If  this  change 
in  attitude  occurred  between  the  control  group  measurement 
and  the  treatment  group  measurement,  it  could  cloud  the  true 
treatment  effect.  To  minimize  this  potential,  the  control 
group  consisted  of  the  last  two  classes  of  SYS-400  before 
the  treatment  was  applied.  This  was  done  to  allow  any 
external  environmental  effects,  not  measured  in  the  survey 
instrument,  to  have  the  same  Impact  on  both  the  control  and 
treatment  groups,  thus  cancelling  the  effect  of  those 
extraneous  variables. 

The  other  minimization  technique  was  to  use  the  same 
instructor  for  both  the  control  and  treatment  groups.  This 
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minimized  the  variability  of  instruction.  The  course 
instructor  was  a  highly  experienced  instructor  who  had 
taught  the  course  many  times  before  this  experiment  began. 
Further,  the  instructor  had  taught  the  decision  analysis 
block  of  Insti'uctlon  twice  before  the  control  group 
instruction.  This  eliminated  the  effect  of  an  instructor 
improving  his  teaching  techniques  significantly,  as  an 
inexperienced  instructor  does,  from  the  control  group  to  the 
treatment  group. 

Instruction .  The  Instruction  on  decision  tree  analysis 
consisted  of  a  reading  assignment,  an  in-class  example  of 
the  calculations  involved,  discussion  of  risk  analysis  and 
sensitivity  analysis  and  the  in-class  exercise  using 
decision  tree  analysis  in  a  hypothetical  case  study.  The 
instruction  and  the  in-class  exercise  lasted  six  hours,  four 
hours  on  a  Friday  and  two  hours  the  following  Monday.  The 
treatment  didn’t  change  the  topics  covered,  the  time  allowed 
or  the  instructor. 

Selection  of  Treatment .  Several  different  computer 
programs  were  evaluated.  The  selection  criteria  were: 

1.  Compatible  with  the  existing  personal  computers 
available  to  the  PCE  class,  a  Zenith  Z-248  or  a  Zenith  PC 
Laptop.  These  type  of  computers  are  commonly  available  at 
the  students  own  work  sites. 

2.  Easy  to  learn,  less  than  one  hour,  so  not  to  detract 
from  learning  decision  tree  analysis. 
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3.  Fulfills  requirements  of  the  exercise  presented  in  the 
current  PCE  curriculum. 

4.  Obtainable  through  existing  DoD  contracts,  public 

domain,  or  cost  less  than  fifty  dollars.  Low  cost  was 
izpcr  ified  *:  j  l?r,pr*"^ve  t’  ;  ch.vr.Cr'a  M.a  students  could 

easily  obtain  the  software  after  returning  to  their 
respective  Jobs. 

•The  computer  decision  support  system  (DSS)  was  not 
limited  to  only  the  technique  that  was  taught.  It  could 
include  several  different  quantitative  decision  support 
techniques.  The  program  was  selected  to  be  easy  to  use 
forthe  computer  novice,  versatile  in  the  type  of  problems 
available  for  analysis,  compatible  with  other  computer 
programs  that  the  student  might  use  at  the  program  office 
and  easy  to  change  parameters  to  answer  ad  hoc  questions. 

The  program  was  selected  three  weeks  prior  to  the 
treatment  group’s  first  day  of  class.  Strategic  Decision 
Group's  Supertree  decision  tree  analysis  program.  Student 
Edition,  was  chosen.  It  was  chosen  because  of  the  menu 
interface,  compatibility  with  popular  spreadsheet  programs 
and  very  large  varied  decision  tree  analyses  were  available 
(35:238,240) . 

The  program  was  installed  on  the  Zenith  PC  Laptop.  A 
menu  was  used  to  select  the  program.  This  isolated  the 
novice  computer  user  from  having  to  learn  the  software 
commands  and  the  operating  system.  The  program  was 
demonstrated  to  the  course's  Instructor.  The  demonstration 
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included  accomplishing  the  in-class  exercise  that  needed  the 
decision  tree  analysis  technique. 

The  Treatment .  This  experiment  was  conducted  on  two 
groups  of  project  managers.  Each  group  was  taught  how  to 
conduct  a  decision  tree  analysis.  The  oontrol  group  (two 
SYS-400  classes)  was  Instructed  on  the  manual  method  of 
using  a  quantitative  decision  support  technique.  The 
treatment  group  was  instructed  on  how  to  use  a  personal 
computer  (PC)  decision  support  system  (DSS) ,  Supertree,  to 
perform  the  same  calculations  required  of  the  decision  tree 
analysis  by  the  control  group. 

No  changes  were  made  in  the  current  method  of 
instruction.  For  both  the  control  and  treatment  groups, 
decision  tree  calculations  and  the  in-class  exercise  were 
explained  in  the  previous  night’s  reading  assignment.  In 
class,  a  decision  tree  analysis  example  was  demonstrated. 

The  in-class  exercise  was  explained  and  the  class  separated 
into  four  teams  to  solve  the  class  exercise. 

Each  member  of  the  treatment  group  was  given  a  user’s 
guide  that  demonstrated  how  to  use  the  software.  The  guide 
explained  only  the  commands  necessary  to  accomplish  the 
decision  tree  analysis  of  the  required  class  exercise.  The 
example  shown  in  the  guide  was  the  example  used  by  the 
instructor,  in  class,  to  teach  decision  tree  analysis.  This 
user’s  guide  la  found  in  Appendix  A. 

The  course  instructor  monitored  the  amount  of  the  time 
both  groups  used  to  accomplish  the  class  exercises  using  the 


decision  tree  analysis  technlq'ii^.  The  Instructor  was 
Interviewed  after  each  of  the  treatment  classy's . 

Comparisons  between  the  control  and  treatment  groups  were 
made.  The  amount  of  time  to  accomplish  various  parts  of  the 
instruction  an-i  tho  in  class  cverclso  were  rreorded.  The 
same  amount  of  time  was  allotted  for  the  Instruction  and  the 
exercise  for  all  of  the  classes.  This  may  be  an  indication 
of  how  easily  the  technique  may  be  used  in  the  program 
manager’s  actual  work  situation. 

Measurement 

The  purpose  of  the  after-course  survey  instrument  was 
to  measure  the  respondents  familiarity  with  and  use  of 
several  quantitative  decision  support  techniques  after 
attending  the  SYS-400  course.  Questions  were  asked  about 
their  perceptions  of  the  possible  benefits  of  computerizing 
decision  support  techniques  and  how  many  techniques  are 
available  on  a  computer.  Demographic  and  responsibility 
area  questions  were  also  asked  to  determine  if  they  might 
have  affected  the  measurement  of  the  treatment,  effect. 

If  the  demographics  varied  considerably  between  the 
control  and  treatment  groups,  then  these  differences  could 
disguise  the  treatment  effect.  The  responsibility  area 
frequency  questions  were  asked  to  determine  the  respondent’s 
program  responsibility.  If  the  respondent  never  made  any 
decisions  affecting  the  program,  the  survey  would  be 
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removed.  It  was  assumed  that  decision  support  techniques 
wouldn’t  be  used  by  a  nondecision  maker  from  this  group. 

The  control  group  and  the  treatment  group  were  surveyed 
three  months  after  course  completion.  The  same  after-course 
survey  was  used  for  both  grouns.  The  survey  was  developed 
from  a  previous  survey  administered  to  a  large  random  sample 
of  Air  Force  program  managers.  That  survey  was  conducted  in 
1985  by  Roger  D.  Koblo ,  a  Master  of  Science  degree  recipient 
of  the  Air  Force  Institute  of  Technology.  This  survey  was 
chosen  because  of  the  high  response  rate  and  the  high 
reliability  of  the  responses  that  were  measured. 

This  survey’s  demographic  information  and  attitude 
measures  of  quantitative  techniques  were  modified  from 
Xoble'a.  Other  demographic  categories,  age  group,  computer 
experience  and  computer  use  at  work  were  added.  The 
modifications  to  the  survey  focused  on  (1)  determining  how 
often  the  subjects  have  used  quantitative  decision  support 
techniques  and  (2)  determining  student  computer  experience. 
These  categories  were  added  to  better  determine  other 
effects  than  the  treatment  effect.  This  after-course  survey 
Is  found  In  Appendix  B. 

Several  survey  construction  techniques  were  used  to 
Increase  the  response  rate,  the  number  of  surveys  completed 
and  returned  versus  the  number  sent  out  {17:165-173).  The 
survey  had  eighty-one  questions  and  was  seven  pages  long, 
less  than  the  proposed  125  1  ten  and  eleven  page  limits 
(17:55).  Surveys  that  are  longer  than  those  limits  tend  not 
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to  be  completed  and  returned.  Upper  case  letters  were  used 
lor  the  answers  and  normal  capitalizations  were  used  for  the 
questions.  This  aided  the  respondent  to  identify  the 
distinction  between  the  questions  and  answers  without  really 
Jiotiei/iei  (17:133,  .  The  printed  question  layout  was 
controlled  sc  that  the  questions  guided  the  respondent  from 
the  top  of  the  page  to  the  bottom  (17:137).  Directions  were 
provided  when  needed  ( 17 : 138)  .  Precisely  worded  close-ended 
questions  with  ordered  answer  choices  were  used. 

This  question  structure  is  ideally  suited  for 
determining  such  things  as  intensity  of  feeling,  degree 
of  Involvement,  and  frequency  of  participation.  This 
type  of  question  uses  the  information  supplied  by 
respondents  to  determine  the  extent  to  which  each 
respondent  differs  from  every  other  one.  Thus 
responses  to  such  questions  are  well  suited  for  many 
lormc  of  sophisticated  analyses  (e.g.,  regression 
analysis). 

The  survey  was  composed  with  the  guidance  of  several 
experts  of  survey  construction  and  several  experts  in  the 
field  of  program  management.  These  experts,  along  with  five 
program  mangers  not  participating  in  the  quasi -experiment , 
were  involved  with  the  testing  of  this  Instrument 
(17:156-157).  The  results  of  the  testing  were  used  to 
modify  the  survey  so  that  it  would  be  clearer  to  the 
respondents  and  more  focused  on  the  objectives  of  this 
research . 

The  cover  letter  was  composed  in  accordance  with  AFIT 
Operating  Instruction  53-10.  The  letter  explained  to  the 
respondent  why  this  after-course  survey  was  Important,  the 
respondents  importance  to  the  study  and  to  minimize  any 
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questions  the  respondent  may  have  had.  This  letter  provided 
the  researcher’s  mailing  address  and  phone  number  should  the 
respondent  need  additional  information.  Confidentiality  was 
assured.  The  cover  letter  gave  instructions  to  request 
study  r«‘^ult.o  should  the  respondent  desire  them.  To 
further  increase  the  response  rate,  the  cover  letter  was 
printed  on  colored  letterhead,  with  specific  dates  and 
personalized  salutations.  The  cover  letter  was  signed  by 
the  course  director  in  blue  ink  and  on  a  soft  surface  to 
leave  a  slight  indentation.  This  left  ‘unmistakable 
evidence  that  the  signature  is  real*  (17:173). 

Several  more  steps  were  taken  to  Increase  the  response 
rate.  All  of  the  contents  were  Inserted  into  the  envelope 
so  everything  would  be  removed  together  and  nothing  would 
accidentally  be  left  in  the  mailing  envelope.  A 
preaddressed  return  envelope  was  provided  (17:178-180).  No 
postage  was  necessary  because  the  military  mail  distribution 
system  was  used.  Because  of  the  small  size  of  each 
experiment  group,  about  twenty-five,  the  researcher 
attempted  to  contact  each  respondent  by  phone.  This  was 
done  the  day  before  and  the  day  of  the  surveys  being  mailed. 
Several  members  were  able  to  provide  address  corrections  and 
forwarding  instructions. 
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Analysis 


The  analysis  focused  on  the  weakness  of  the  experiment 
design,  nonequivalent  groups.  The  analysis  started  with 
theminlmum  number  of  assumptions  about  the  responses’  data 
levels  and  distributions.  Two  assumptions,  the  categorical 
(nominal)  data  level  and  the  unknown  distributions  of  the 
responses,  meant  nonparametr ic  statistical  tests  should  be 
used.  Since  some  of  the  responses  were  written  on  a  Likert 
scale,  parametric  teats  could  also  be  used  in  the  analysis. 

A  Likert  scale  la  ordinal  data  but  la  sometimes  considered 
interval  if  equal  intervals  between  the  answers  are  assumed 
(10)  . 

All  of  the  tests  used  in  this  analysis  provided  a 
probability  that  the  sample  distributions  were  similar.  If 
the  probability  was  lower  than  alpha,  then  the  distributions 
could  be  considered  different.  Alpha  la  the  test  percentage 
error  that  the  researcher  is  willing  to  except,  of 
determining  that  the  samples  are  dissimilar  when  they 
actually  aren't  (39: 138-139) .  The  testa  are  not  conclusive 
on  why  the  difference  exists,  but  only  that  there  is  a 
d i f  f  erence . 

The  respondents  answers  to  the  demographic  and 
responsibility  area  questions  were  analyzed  first.  The 
surveys  were  separated  into  the  control  and  treatment  groups 
and  compared.  This  helped  determine  if  the  control  and 
treatment  groups  were  from  the  same  larger  population.  If 
the  demographics  and  responsibility  areas  varied 
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significantly,  then  few  conclusions  could  be  drawn  about  the 


effect  of  the  treatment. 

The  respondent’s  demographic  and  responsibility  area 
categories  might  have  caused  the  type  of  response  on  the 
different  decision  support  techniques  to  vary.  Table  1 
shows  the  categories  to  questions  one  through  eleven  and  the 
hypothesized  influence  on  frequency  of  technique  use  that 
might  have  disguised  the  effect  of  the  treatment. 


Table  1 


Hypothesized  Demographic  Influence  on  Frequency  of  Use 


*  Question’s  Subject 

Hypothesized  Influence  on  Use 

1.  Product  Division 

Unknown 

2 .  Rank 

+ 

3.  Academic  Degree 

+ 

4.  Field  of  Study 

-  Busi/Oth  +  Eng/Sci 

5.  SPO  Type 

Unknown 

6.  Job  Position 

- 

7.  #  Yrs  Experience 

+ 

8 .  Age 

+ 

9.  Acquisition  Phase 

+  /- 

10.  Computer  Familiar 

+ 

1 1 .  Computer  Work  Use 

+ 

12.-31.  Responsibility 

Unknown 

The  responses  to  the  main  responsibility  areas  were 


compared  to  the  responses  to  the  specific  responsibility 
areas.  This  was  done  to  test  the  consistency  of  the 
respondents'  answers.  The  higher  the  reliability,  the  more 
consistent  the  responses.  The  responses  to  quantitative 
decision  support  technique  familiarity  and  frequency  of  use 
were  compared  next.  The  possible  effect  of  dlfferin,^ 
demographics  on  the  responses  to  each  of  the  decision 
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support  techniques  was  then  examined.  Results  of  the 


computerized  benefits  and  computer  program  availability  were 
then  tabulated  and  reported. 

Tests .  If  the  survey  contained  obvious  mistakes,  It 
was  removed  from  the  analysis.  The  responses  to  questions 
seventy-two  through  elghty-one,  the  two  optional  "OTHER' 
quantitative  decision  support  techniques,  were  combined  with 
the  appropriate  category  from  questions  thirty-two  through 
seventy-one.  Any  survey  response  that  was  not  answered,  was 
not  used  in  a  test. 

Alpha  analysis  was  used  to  teat  the  reliability  and 
validity  of  the  respondent’s  answers  to  the  responsibility 
area  survey  questions,  question  twelve  through  thirty-one. 
Five  main  areas  were  asked  In  questions  twelve  through 
sixteen.  Questions  seventeen  through  thirty-one  asked  more 
specific  responsibility  areas  in  the  five  main  areas.  The 
respondents’  answers  were  tested  to  see  if  they  answered 
similarly  to  questions  that  related  to  each  of  the  main 
areas.  This  tested  the  reliabii’.ty  and  consistency  of  the 
respondent’s  answers  in  each  of  the  five  areas. 

All  of  the  responses  were  compared  with  a  Chi  squared 
(X^)  test  of  homogeneity,  a  test  for  similarities  between 
separate  groups.  This  was  used  to  determine  whether  the 
respondents  could  be  from  the  same  population.  The  test 
determines  "whether  the  relative  frequency  distribution  [of 
the  responses  compared  between  the  groups]  are  the  same" 
{3a!4.3) .  This  test  was  used  to  determine  if  the  control 
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and  treatment  groups  demographics  were  different  from  each 
other.  The  test  doesn’t  assume  a  specific  population 
distribution.  The  test  compares  the  responses  of  each  group 
and  testa  If  the  responses  are  related  to  the  control  versus 
treatment  category  or  if  they  could  be  from  the  same  overall 
group . 

A  test  more  sensitive  to  differences  in  the 
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distributions  of  two  samples  than  the  X  test  is  the 
Wllcoxson  Rank  Sum  test.  The  additional  assumption  from  the 
X  test  is  the  data  level  has  to  be  ordinal  or  higher. 

Each  of  the  demographic  questions  were  examined  to  see 
if  the  range  of  responses  for  each  category,  in  the 
demographic  question,  resulted  in  similar  distributions  of 
responses  to  the  questions  for  each  technique  category.  The 
Kruskal -Wal 1  is  one  way  analysis  of  variance  was  used. 

Kruskal -Wal 1  is  test  is  a  generalization  of  the 
rank-sum  teat.  Tied  values  are  assigned  their  average 
rank.  If  each  of  the  groups  had  similar  distributions, 
the  mean  rank  for  all  groups  would  be  expected  to  be 
■similar' .  (38:6. 18) 

If  respondents  of  one  or  more  demographic  category  tended  to 
respond  differently  from  the  other  categories,  then  this 
could  effect  the  results  of  the  analysis  of  the  treatment. 
The  observed  effect  of  the  treatment  might  be  clouded  by  the 
effect  of  a  demographic  category. 

A  test  more  sensitive  to  differences  than  the 
X^  and  the  Rank  Sum  is  the  Student's  t  test.  The  Student’s  t 
statistical  test  was  used  to  compare  the  survey  responses  of 
the  questions  that  could  be  considered  Interval  data.  A 
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Student’s  t  test  was  used  since  the  actual  mean  and  standard 
deviation  of  the  overall  population  was  unknown.  The 
Student’s  t  test  assumes  that  the  responses  were  obtained 
independently  and  that  the  sample  was  from  a  range  of 
responses  from  an  overall  population  whose  responses  would 
be  mound  shaped  distributed  (39:174-175).  If  there  is  a 
difference,  the  probability  will  be  less  than  the  confidence 
level  probability  used  for  comparison,  alpha. 

Summary 

To  test  the  hypothesis,  the  frequency  of  using 
quantitative  decision  techniques  changes  if  a  technique  is 
taught  with  the  aid  of  a  computerized  decision  support 
system.  A  posttest-only  design  with  nonequivalent  groups 
quasi -experiment  was  conducted.  The  groups  were  selected 
from  the  Intermediate  Program  Management  (SYS-400) 
Professional  Continuing  Education  (PCE)  course  at 
Wr ight-Patterson  AFB  OH.  The  course  teaches  decision  tree 
analysis.  The  treatment  was  to  introduce  the  decision 
analysis  with  the  aid  of  Strategic  Decision  Group's 
Supertree  program.  Student  Edition.  A  after-course  survey 
was  developed  to  measure  the  effects  of  the  treatment. 
Several  techniques  were  used  to  Increase  the  survey  response 
rate.  The  analysis  started  with  as  few  assumptions  as 
possible.  Tests  were  conducted  to  determine  the  control  and 
treatment  groups’  similarities  and  differences.  Higher 
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level  tests  were  used  only  when  the  assumptions  were  valid. 
The  results  of  the  analysis  are  In  chapter  IV. 
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I V .  Reaul ta 


Introduction 

Thla  chapter  addreaaea  the  reaulta  of  the 
quaa i -exper iment  deacribed  in  chapter  III.  The  hypotheala 
teated  waa  if  program  and  project  managers  are  taught  to  uae 
a  computer  to  perform  the  difficult  computations  of  a 
quantitative  decision  support  technique,  they  are  mot-e 
likely  to  apply  that  technique,  and  possibly  others,  to 
actual  problems  that  occur  in  their  respective  jobs.  These 
results  were  measured  by  the  responses  from  the  three  month 
after-course  survey.  The  analysis  attempted  to  determine 
the  effect  of  the  treatment  and  to  identify  any  variables 
that  might  disguise  the  treatment  effect.  The  conclusions 
from  this  analysis  and  recommendations  for  future  research 
are  presented  in  chapter  V. 

In-Class  Exercise 

The  Intermediate  Program  Management  (SYS-400) , 
Professional  Continuing  Education  (PCE)  course  at 
Wr ight-Patterson  AFB  OH,  classes  that  were  used  in  this 
study  are  listed  in  Table  1.  The  group  class  dates, 
exercise  dates  and  number  in  each  class  that  completed  the 
curriculum  are  shown. 
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Table  2 


Experiment  Group  Information 


Group 

SYS-400 

Dates 

Exercise 

Dates 

Group 

Size 

Contro 1 

89A 

17-28  Oct  88 

21-24  Oct  88 

24 

89B 

5-16  Dec  88 

9-12  Dec  88 

24 

Treatment 

89C 

23  Jan  -  3  Feb  89 

27-30  Jan  89 

23 

89D 

21  Feb  -  3  Mar  89 

24-27  Feb  89 

24 

The  Instructor  stated  that,  as  a  whole,  the  control 
group  teams  had  gotten  tired  by  the  end  of  the  manual 
decision  tree  calculations  and  had  lost  their  motivation  to 
do  much  sensitivity  and  risk  analysis  in  the  in-class 
exercise.  The  exercise  was  laborious,  with  many  manual 
mathematical  calculations.  None  of  the  control  group  teams 
were  finished  by  Friday  afternoon.  Many  teams  felt 
exhausted  and  elected  to  stop  thirty  to  forty-five  minutes 
early.  They  completed  the  rest  of  the  calculations  on 
Monday  morning. 

The  treatment  group  teams  were  observed  voluntarily 
staying  an  average  of  50  minutes  longer  than  the  control 
group  teams  and  to  be  in  better  spirits  during  and  after  the 
exercise.  The  majority  of  the  treatment  group  teams 
finished  on  Friday  afternoon  with  only  a  few  minor  points 
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left  to  complete  Monday  morning.  Therefore,  the  treatment 
group  teams  had  more  discussion  time  available  on  Monday. 

The  Instructor  stated  that  the  treatment  groups  were 
able  to  receive  a  more  detailed  discussion  on  the  in-class 
exercise  and  the  lesson.  The  time  to  complete  the  in-class 
exercise  was  difficult  to  measure  because  the  treatment 
groups  developed  overhead  slides  and  refined  their  output’s 
appearance.  By  direct  observation,  the  impression  of  the 
instructor  and  this  researcher  is  that  the  treatment  group 
teams  completed  the  same  tasks  as  the  control  group  teams 
in  less  time.  Only  two  to  three  members  on  each  team  in  the 
treatment  group  actually  used  the  computer  to  do  the 
computations  that  were  required.  This  was  comparable  to  the 
control  groups  where  only  one  or  two  members  per  team  did 
the  manual  calculations. 

The  end-of -course  exam  did  not  show  any  significant 
differences  in  the  test  scores  on  decision  tree  knowledge. 
Both  groups  showed  a  high  degree  of  understanding,  85/1. 

The  end-o f -course  critiques  were  similar  between  the 
groups,  although  some  did  comment  positively  on  the 
computer  based  exercise.  The  only  drawback  reported  by  the 
students  was  the  amount  of  time  the  program  took  to  complete 
the  required  calculations.  Even  though  it  was  much  faster 
than  computing  the  analysis  manually,  it  appeared  alow  to 
the  user. 
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The  Survey  Reaul ta 


The  after-aourae  aurveya  were  mailed  thirteen  work 
weeka  after  courae  completion.  Table  3  ahowa  the  achedule 
uaed.  Thia  achedule  waa  choaen  becauae  of  the  Thankagivlng 
and  Chrlatmaa  holidaya.  Work  weeka  were  uaed  to  determine 
the  mailing  time.  Thia  allowed  each  group  to  have  the  aame 
amount  of  time  to  have  program  problema  that  declaion 
aupport  techniquea  could  be  uaed  to  aid  aolvlng  after 
returning  to  their  reapective  Joba. 


Table  3 

Survey  Timing  and  Reaponae  Ratea 


Group 

Survey 

Dates 

Response 

Size 

Response 

Rate 

Control 

89A 

13  Feb  89 

22 

92X 

89B 

13  Mar  89 

19 

79Z 

Treatment 

89C 

8  May  89 

74Z 

89D 

5  Jun  89 

18 

757. 

Reaponae  Rate .  Surveya  were  mailed  to  all  ninety-five 
aubjecta,  forty-eight  to  the  control  group  and  forty-aeven 
to  the  treatment  group.  Telephone  contact  with  all  aubjecta 
waa  attempted  on  the  day  before  and  the  day  the  after  survey 
waa  mailed.  If  the  subject  couldn’t  be  contacted  in  those 
two  days,  a  message  waa  left.  Eighty-five  percent  and 
seventy-four  percent  of  the  control  and  treatment  group 
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surveys  were  returned,  respectively.  Since  the  response 
rate  was  above  the  needed  sixty  percent,  no  further  attempt 
to  contact  the  subjects  were  made. 

All  but  one  of  the  surveys  was  determined  to  be  valid. 
This  was  a  treatment  subject  survey  which  selected  the  first 
response,  1,  for  all  of  the  questions,  except  for  the 
demographics.  The  remarks  in  this  survey  stated  that  the 
respondent  didn’t  work  in  a  program  office;  therefore,  all 
of  the  questions  were  “Not  Applicable*.  Consequently,  this 
survey  was  removed  from  the  analyses. 

Demographics .  The  demographic  characteristics  of  both 
groups  are  shown  in  Table  4.  The  responses  of  both  groups 
were  distributed  similarly  across  all  of  the  demographic 
questions.  Aeronautical  Systems  Division  was  the  dominate 
product  division  represented.  Over  80%  had  completed  a 
graduate  ■’•••riculum.  Approximately  half  of  the  fields  of 
study,  both  undergraduate  and  graduate,  were  in  engineering, 
science  and/or  math.  Over  i0%  were  in  business  and/or 
management.  Over  60%  worked  in  a  system  program  office. 
About  half  stated  that  they  are  currently  program  or  project 
managers.  All  phases  in  the  acquisition  process  were 
represented.  More  than  three  quarters  felt  that  they  are  at 
least  moderately  familiar  with  personal  computers  and/or 
mainframe  computers.  Nine  out  of  ten  use  a  computer  every 
day  . 
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TaMe  4 

i*x'equency  Counts  of  Responses  to  Demographics 


CONTROL  TREATMENT 


QUESTION 

RESPONSE  FREQUENCY 

X  FREQUENCY  % 

1 . PRODUCT 

ESD 

1 

2.4 

0 

0 . 0 

DIVISION 

ASD 

25 

61.0 

21 

61.8 

SD 

0 

0 . 0 

1 

2 . 9 

OTHER 

15 

36.6 

12 

35 . 3 

2. RANK  CAPT  &  BELOW 

6 

14.6 

7 

20 . 6 

MAJOR 

13 

31.7 

16 

47. 1 

LTC  &  ABOVE 

14 

34. 1 

4 

11.8 

aM/GS-12  &  LOW 

2 

4 . 9 

1 

2.9 

GM/QS-13  &  ABV 

6 

14.6 

6 

17 . 7 

3 .HIGHEST 

BACHELORS 

5 

12.2 

6 

17 . 6 

DEGREE 

MASTERS 

35 

85 . 4 

26 

76 . 5 

DOCTORAL 

1 

2.4 

1 

2 . 9 

NONE 

0 

0 . 0 

1 

2 . 9 

4. FIELD  ENGINEERING 

22 

39.3 

18 

36.7 

OF  SCIENCE/MATH 

10 

17.9 

6 

12.2 

STUDY 

BUSI/MNGNT 

23 

41.1 

21 

42.9 

NONE  OF  ABOVE 

1 

1.8 

4 

8.2 

5.CPO 

BASKET 

10 

24.4 

10 

29 . 4 

TYPE 

SINGLE  SYS 

16 

39.0 

13 

38.2 

OTHER 

15 

36 . 6 

1  1 

32 . 4 

6.  JOB 

DIRECTOR 

0 

0.0 

0 

0 . 0 

POSITION 

PGM  MNGR 

1  1 

26 . 8 

13 

38 . 2 

PHJT  MNGR 

6 

14.6 

5 

14.7 

OTHER 

24 

58 . 5 

16 

47.  1 

7 . C  UMUL AT I VE  NONE 

5 

12.2 

2 

5 . 9 

EXPERIENCE  1  YR 

2 

4 . 9 

3 

8 . 8 

2-3  YRS 

1 1 

26 . 8 

7 

20 . 6 

4-6  YRS 

12 

29 . 3 

9 

26 . 5 

ABV  6  YRS 

1  1 

26 . 8 

13 

38.2 
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Table  4  (Continued) 


Frequency  Counts  of  Responses  to  Demographics 


CONTROL 

TREATMENT 

QUESTION 

RESPONSE 

FREQUENCY 

V.  FREQUENCY 

% 

8.  AGE 

UNDER  30 

1 

2.4 

1 

2.9 

GROUP 

30  -  35 

8 

19 . 5 

14 

41.2 

36  -  40 

17 

41.5 

12 

35 . 3 

41  -  45 

10 

24.4 

5 

14.7 

OVER  45 

5 

12 . 2 

2 

5 . 9 

9.ACQ 

NO  ANSWER 

3 

N/A 

8 

N/A 

PHASE 

CNCPT  EXPLR 

8 

14.3 

1 1 

18 . 6 

DEM/VAL 

9 

16. 1 

15 

25 . 4 

FULL  SCALE 

20 

37.5 

19 

32 . 2 

PROD  &  DEPL 

18 

32 . 1 

14 

23 . 7 

10. COMPUTER  NONE 

2 

4.9 

1 

2 . 9 

FAMILIARITY  SLIGHTLY 

8 

19.5 

2 

5 . 9 

MODERATELY 

16 

39.0 

19 

55 . 9 

VERY 

7 

17. 1 

7 

20 . 6 

COMPLETELY 

8 

19.5 

5 

14 . 7 

1 1 . COMPUTER  NEVER 

3 

7 . 3 

2 

5.9 

USE 

<  1  HR/ DAY 

12 

29.3  . 

9 

26.8 

1-2  HR/DAY 

16 

39.0 

13 

38 . 2 

3-4  HR/DAY 

7 

17. 1 

5 

14.7 

>  4  HR/ DAY 

3 

7 . 3 

5 

14.7 

^  Test .  The  test 

of  the 

demographic  questions 

is  shown 

i n  Tab 1 e  5 . 

The  test 

suggests 

that  the 

sample 

populations  are  from 

the  same  overall  population 

for  values 

of  alpha 

up  to  0.05. 

The  only 

demographic  category  that 

could  be 

rejected,  if 

one  uses 

an  alpha 

of  0.10, 

as  coming 

from  the 

same  population  is  Age 

Group . 

An  alpha 

of  0.10  was 

selected  because  the  sample  population,  ninety-five,  is 


small  when  compared  to  the  entire  system  acquisition  program 
manager  population.  The  hypothesis  tested  was  that  the 
sample  populations  were  from  the  same  overall  population. 
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The  alternate  hypothesis  was  that  the  sample  population  are 


somehow  different.  There  was  no  assumption  of  the 
population  distribution.  The  data  level  required  for  this 
test  is  categorical  or  higher.  The  initial  conclusion  is 
that  the  demographics  are  similar  enough  to  conclude  that 
both  groups  belong  to  the  same  larger  population.  This 
conclusion  means  that  any  differences  in  other  variables 
might  be  attributed  to  the  effect  of  the  treatment. 


Table  5 

Besults  of  Comparison  between  Control 
and  Treatment  Groups’  Demographics 


Question 

X= 

m 

Probabi 1 i ty 

1  . 

Product  Division 

0.013 

1 

0.908 

2. 

Rank 

5.672 

4 

0 . 225 

3. 

Highest  Degree 

0.974 

1 

0.324 

4  . 

Field  of  Study 

2 . 837 

3 

0.416 

5. 

SPO  Type 

0.275 

2 

0 . 872 

6. 

Job  Position 

1.215 

2 

0.545 

7  . 

Cumulative  Experience 

1  .  174 

3 

0 . 579 

8  . 

Age  Group 

4.658 

2 

0 . 097 

9. 

Acquisition  Phase 

2 . 362 

3 

0.501 

10. 

Computer  Familiarity 

4.101 

3 

0.251 

1 1  . 

Computer  Use 

0 . 275 

2 

0 . 872 

Wi  1 coxon  Rank  Sum  Tes t .  The  Wllcoxon  Rank  Sum 
test  of  the  demographic  questions  is  shown  in  Appendix  C. 

As  with  the  test,  this  test  suggests  that  the  sample 
populations  are  from  the  same  overall  population  for  values 
of  alpha  up  to  0.05.  The  only  demographic  category  that 
might  be  rejected  as  coming  from  the  same  population  is  AGE 
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GROUP,  if  one  accepts  an  alpha  of  0.10.  There  was  no 
assumption  of  the  population  distribution.  The  data  level 
required  for  this  test  is  ordinal  or  higher.  The  only 
question  that  can’t  translate  into  ordinal  data  is  PRODUCT 
DIVISION.  The  test  for  PRODUCT  DIVISION  would  still  show  a 
tendency  that  the  sample  populations  might  be  different  if 
the  probability  was  low.  Since  the  probability  is  very 
high,  the  conclusion  is  that  the  respondents  came  from  the 
same  overall  population. 

The  conclusion  from  both  testa  is  that  the  demographics 
are  similar  enough  to  conclude  that  both  groups  belong  to 
the  same  larger  population.  The  AGE  GROUP  category  was 
investigated  later,  with  the  Kruskal -Wal 1  is  test,  for 
effects  on  the  treatment.  That  means  any  differences  in  the 
other  variables  might  be  attributed  to  the  effect  of  the 
treatment . 

Res  pons i b i I i ty  Areas . 

Rel iabi 1 i ty  Tes t .  A  reliability  teat  was  used  to 
test  for  consistency  in  the  respondents’  answers.  Each  of 
the  five  major  area  answers,  questions  twelve  through 
sixteen,  were  compared  to  the  answers  of  the  specific  area 
response,  questions  seventeen  through  thirty-one.  The  major 
area  responses  were  then  compared  with  the  questions  that 
were  included  to  test  reliability.  The  results  are  shown  in 
Table  6.  The  very  high  reliability  scores  show  the 
respondents  were  consistent  in  their  T^csponses  for  questions 
twelve  through  thirty-one.  This  leads  to  the  conclusion 
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that  each  respondent  was  trying  to  answer  the  survey 
questions  as  thoroughly  as  they  could. 


Table  6 

Survey  Reliability  Test  Results 


Question  Range 

Reliability  Alpha 

12,  17-31 

0 . 8820 

12.  20,  27.  31 

0.7110 

13.  17-31 

0.8869 

13.  18.  26.  28 

0 . 8365 

14.  17-31 

0 . 8861 

14,  22.  25,  29 

0.7647 

15,  17-31 

0 . 8844 

15,  17,  21,  23.  30 

0 . 7893 

16.  17-31 

0  8798 

16,  19.  24 

0.8314 

Since  the  responses  were  so  consistent,  only  the  responses 
to  questions  twelve  through  sixteen  were  used  to  analyze  the 
responses  to  responsibility  areas. 

Test .  The  X^  test  shows  that  both  the  control 
group  and  the  treatment  group  have  similar  responsibilities 
in  their  program  office.  Only  the  responses  in  the 
LOGISTICS  area  approached  being  significantly  different. 
Appendix  D  shows  the  results  of  the  teats  for  the  main 
responsibility  areas  and  the  results  for  the  more  specific 
responsibility  areas,  grouped  by  main  area. 
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The  conclusion  from  this  analysis  Is  that  the  control 
and  treatment  groups  have  similar  levels  of  responsibility 
in  their  program  office  and  in  the  same  areas,  except 
possibly  in  the  LOGISTICS  area. 

Wi  Icoxon  Rank  Sum  Test .  The  Wilcoxon  Rank  Sum 
test  showed  that  both  the  control  group  and  the  treatment 
group  responses  were  similar.  Again,  the  only  difference 
appeared  in  the  LOGISTICS  area,  when  compared  to  alpha  of 
0.10.  The  results  are  shown  in  Appendix  E. 

This  confirms  the  analysis.  Both  the  control  and 
treatment  groups  had  similar  levels  of  responsibility  in 
their  program  office  and  in  the  same  areas,  again  except 
in  the  LOGISTICS  area. 

Student ’ s  ^  Test .  The  Student’s  t  test  confirmed 
that  the  sample  pooulatlons  are  for  a  similar  overall 
population.  The  results  are  shown  in  Table  7.  All  three 
tests  confirmed  each  other.  In  the  treatment  group,  more 
decisions  in  the  LOGISTICS  area  were  made.  This  shows  that 
the  data  level  assumption  was  acceptable  and  the  tests  had  a 
high  degree  of  reliability. 

The  conclusion  from  these  analyses  is  that  both  the 
control  and  treatment  groups  had  similar  levels  of 
responsibility  in  their  program  office  and  in  the  same 
areas,  except  possibly  in  the  LOGISTICS  area.  The  LOGISTICS 
area  was  analyzed  later,  with  the  Kruskal -Wal 1 1 s  test,  for 
effects  on  the  treatment  variables. 
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Table  7 

Student’s  t  Test  Results  of  Comparison  between 
Control  and  Treatment  Groups’  Responsibility  Areas 


Question 

t 

df 

Probabi 1 i ty 

12.  Cost  and  Budget 

-  1 . 345 

71.4 

0 . 1829 

13.  Schedule 

-0.035 

72 . 8 

0.9721 

14.  Technical  Performance 

0 . 164 

71.2 

0 . 8701 

15.  Logistics 

-2 . 238 

70.7 

0 . 0284 

16.  Admin  of  Pgm  Personnel 

1.163 

70 . 3 

0 . 2487 

Quantitative  Peel s 1 on  Support  Techniques .  A  alpha  of 
0.10  was  selected  because  the  sample  population, 
ninety-five,  was  small,  forty-one  and  thirty-four,  when 
compared  to  the  entire  system  acquisition  program  manager 
population.  Any  probability  less  than  0.10  meant  that  the 
distributions  were  not  similar. 

Test .  The  X^  test  was  used  to  determine  if 
there  were  any  differences  between  the  control  and  treatoient 
group’s  responses  without  assuming  a  high  data  level  or 
response  distribution.  The  results  are  presented  grouped  by 
quantitative  decision  support  technique  category. 

The  test  showed  that  the  control  and  treatment  groups 
were  different  on  decision  support  technique  Familiarity  for 
the  Statistical  Analysis,  Linear  Programming,  Simulation, 
Proje.'t  Selection  and  Financial  Methods  categories. 

The  teat  showed  that  the  control  and  treatment  groups 
were  different  on  the  technique  Use  frequency  for  the 
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Statistical  Analysis,  Linear  Programming  and  Financial 
Methods  categories. 

The  test  showed  that  the  control  and  treatment  groups 
were  different  on  the  technique  computerization  Benefit  for 
the  Linear  Programming,  Simulation,  Project  Selection  and 
Financial  Methods  categories. 

The  test  showed  that  the  control  and  treatment  groups 
were  different  on  the  technique  computer  program 
Availability  for  the  Statistical  Analysis  and  Financial 
Methods  categories. 

The  test  showed  that  the  control  and  treatment  groups 
were  different  on  the  technique  use  Opportunity  only  for  the 
Financial  Methods  category. 

The  detailed  test  results  are  in  Appendix  F.  Table  8 
summarizes  the  categories  that  were  different  by  technique 
question  area.  The  X  means  a  difference  between  the  control 
and  treatment  groups . 

Wi  1 coxon  Rank  Sum  Tes t .  In  the  Wilcoxon  Rank  Sum 
test  for  technique  Familiarity,  Statistical  Analysis,  Linear 
Programming,  Simulation,  Project  Selection  and  Financial 
Methods  categories  were  different  between  both  groups. 

These  results  agreed  with  the  test  results. 
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Table  8 


Summary  of  Differences  found  Through  Test  of 
Comparison  between  Control  and  Treatment  Groups  to  the 
Quantitative  Decision  Support  Technique  Questions 


Category 

Technique  Question  Area 

Familiar 

Dse 

Benef it 

Available 

Opportuni t> 

Statistical  Analysis 

X 

X 

X 

Linear  Programming 

X 

X 

X 

Simulation 

X 

X 

Project  Selection 

X 

X 

Decision  Theory 

Ranking 

Networking 

Financial  Methods 

X 

X 

_ 

X 

X 

X 

X  means  a  Difference  between  the  Control  and  Treatment  Groups 


The  test  for  technique  Use  frequency.  Statistical 
Analysis  and  Financial  Methods  categories  were  different 
between  both  groups.  Linear  Programming  did  not  test  as 
being  different  as  in  the  test.  This  could  mean  that 
there  wasn’t  a  treatment  effect  on  Linear  Programming 
decision  support  technique. 

The  test  for  technique  computerization  Benefit, 
Financial  Methods  category  was  different  between  both 
groups.  Linear  Programming  and  Simulation  did  not  test  as 
being  different  as  in  the  X^  test.  This  could  mean  that 
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there  wasn’t  a  treatment  effect  on  Linear  Programming  and 
Simulation  decision  support  techniques. 

The  test  for  technique  computer  program  Availability, 
Financial  Methods  category  was  different  between  both 
groups.  Statistical  Analysis  did  not  test  as  being 
different  as  In  the  test.  This  could  mean  that  there 
wasn’t  a  treatment  effect  on  Statistical  Analysis  decision 
support  technique. 

The  test  for  technique  use  Opportunity  of  Financial 
Methods  category  was  different  between  both  groups.  These 
results  agreed  with  the  test  results. 

The  detailed  demographic  test  results  are  In 
Appendix  G.  Table  9  summarizes  the  categories  that  were 
different  by  technique  question  area.  The  X  means  a 
difference  between  the  control  and  treatment  groups. 

Student ’ s  ^  Test .  The  Student  t  test  showed  that  the 
control  and  treatment  groups  were  different  on  decision 
support  technique  Familiarity  for  the  Statistical  Analysis, 
Linear  Programming,  Simulation,  Project  Selection  and 
Financial  Methods  categories.  The  detailed  results  are  In 
Appendix  H. 

The  means  and  standard  deviations  for  the  responses  are 
listed  In  Appendix  I.  The  treatment  group  averages  were  all 
higher . 
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Table  9 


Summary  at  Differences  found  Through  Hank  Sum  Test  of 
Comparison  between  Control  and  Treatment  Groups  to  the 
Quantitative  Decision  Support  Technique  Questions 


Category 

Technique  Question  Area 

Familiar 

Use 

Benef it 

Available 

Opportunity 

Statistical  Analysis 

X 

X 

Linear  Programming 

X 

Simulation 

X 

Project  Selection 

X 

Decision  Theory 

Ranking 

Networking 

Financial  Methods 

X 

X 

X 

X 

X 

X  means  a  Difference  between  the  Control  and  Treatment  Groups 


Statistical  Analysis,  Linear  Programming  and  Financial 
Methods  were  different  for  Use  frequency. 

The  test  showed  that  the  control  and  treatment  groups 
were  different  on  the  technique  computerization  Benefit  for 
the  Linear  Programming,  Simulation,  Project  Selection  and 
Financial  Methods  categories. 

The  control  and  treatment  groups  were  different  on  the 
technique  computer  program  Availability  for  the  Statistical 
Analysis  and  Financial  Methods  categories. 
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The  test  measured  the  control  and  treatment  groups 
difference  on  the  technique  use  Opportunity  only  for  the 
Financial  Methods  category. 

Table  10  summarizes  the  categories  that  were  different 
by  technique  question  area.  The  X  means  a  difference 
between  the  control  and  treatment  groups. 

This  leads  to  the  conclusion  that  the  treatment  did 
cause  the  differences  between  the  two  groups  for  those 
categories . 

Table  10 

Summary  of  Differences  found  Through  Student’s  t  Test  of 
Comparison  between  Control  and  Treatment  Groups  to  the 
Quantitative  Decision  Support  Technique  Questions 


Category 

Technique  Question  Area 

Familiar 

Dse 

Renef i t 

Avai 1 abl e 

Opportiini  ty 

Statistical  Analysis 

X 

X 

Linear  Programming 

X 

X 

Simulation 

X 

Project  Selection 

X 

X 

Decision  Theory 

Ranking 

Metworking 

Financial  Methods 

X 

X 

X 

X 

X 

X  means  a  Difference  between  the  Control  and  Treatment  Groups 
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Kruakal -Wal 1 la  Teat .  The  Kpuakal -Wal 1 i a  teat 
examined  the  dlatrlbutlon  of  reaponaea  by  each  demographic 
reaponae  category  and  the  LOGISTICS  reapona 1 bl 1 1 ty  area 
varlablea.  The  aurveya  were  not  aeparated  Into  control  and 
treatment  groupa.  When  the  diatr Ibutiona  acroaa  all 
variable  reaponaea  are  almllar,  that  auggeats  the  variable 
did  not  effect  the  reaponaea  to  the  other  queationa,  like 
frequency  of  technique  Uae.  Thr  varlablea  that  were  not 
diatributed  aimllarly  acroaa  the  range  of  reaponaea  might 
have  diaguiaad  the  treatment  effect. 

The  obaerved  treatment  effecta  for  Familiarity  were  in 
Statiatlcal  Analyaia,  Linear  Programming,  Project  Selection 
and  Financial  Methoda.  Statiatlcal  Analyaia,  Linear 
Programming  and  Financial  Methoda  were  different  for 
technique  Uae.  Project  Selection  waa  different  in  technique 
Availability.  The  Financial  Methods  technique  was  also 
different  for  technique  Benefit,  Availability  and  uae 
Opportunity.  These  techniques  were  specifically  examined 
for  variable  effects  other  than  the  treatment.  All  of  the 
variable  effects  for  all  of  the  techniques  are  reported  for 
a  better  understanding  of  the  variable  influence. 

The  demographic  category,  AGE  GROUP,  and  responsibility 
area,  LOGISTICS,  were  the  only  variables  different  between 
the  control  and  treatment  groupa.  AGE  GROUP  waa  diatributed 
evenly  across  the  range  of  all  responses,  except  one.  AGE 
GROUP  did  have  a  difference  for  technique  computerization 
Benelit  in  Pr*ojecL  St?leeLion.  Those  respondents  under* 
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thirty  and  over  forty-five  marked  the  Benefit  higher  than 
the  other  AGE  GROUP  respondents.  This  meant  that  despite 
the  differences  between  the  control  and  treatment  group’s 
distributions  for  AGE  GROUP,  it  had  no  measurable  treatment 
effect  on  Use  frequency. 

The  responsibility  category,  LOGISTICS,  was  not 
distributed  evenly  across  the  range  of  responses.  For  the 
treatment  effected  techniques.  Project  Selection,  Financial 
Methods  Familiarity  LOGISTICS  had  a  positive  effect.  This 
meant  low  levels  of  LOGISTICS  responsibility  marked  low 
Financial  Methods  Familiarity.  The  high  levels  of  LOGISTICS 
responsibility  marked  high  Financial  Methods  Familiarity. 
Linear  Programming  and  Financial  Methods  were  treatment 
effected  techniques.  The  technique  change  might  have  been 
attributed  to  the  Logistics  variable  and  not  the  treatment. 

Table  11  shows  the  technique  Familiarity  differences 
and  the  influences  for  all  of  the  variables.  For  Tables  11 
through  15,  a  plus  or  minus  is  the  direction  the  variable 
category  influenced  the  decision  support  technique  variable. 
The  minus  shows  a  lower  technique  variable  response  and  the 
plus  means  a  higher  technique  variable  response.  For 
example,  the  plus  in  COMPUTER  USE  meant  a  lower  category 
response  marked  lower  technique  response.  The  minus  in 
ACQUISITION  PHASE  meant  that  the  earlier  a  program  is  in  the 
acquisition  cycle,  the  higher  the  technique  response. 
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Table  11 


Kruskal -Wal 1 is  Test  for  Influence  of  Demographic  and 
Logistic  Categories  on  Technique  Familiarity 


•  Question’s  Subject 

Observed  Influence 
on  Familiarity 

Statistical 
10.  Computer  Familiarity 

.  Analysis 

+ 

Linear  Prc 

2 .  Rank 

5.  SPO  Type 

6.  Job  Position 

9.  Acquisition  Phase 

10.  Computer  Familiarity 

11.  Computer  Use 

Jgrammi ng 

-  SPO  +  Other 

-  Pgm  +  Other/Pj  t 

+ 

+ 

Simula 

9.  Acquisition  Phase 

10.  Computer  Familiarity 

it  i  on 

+ 

Project  I 

15.  Logistics 

Selection 

+ 

Decisioi 

3.  Highest  Degree 

10.  Computer  Familiarity 
15.  Logistics 

1  Theory 

+ 

+ 

+ 

Rani 

9.  Acquisition  Phase 

15.  Logistics 

cing 

+ 

Netwoi 

2 .  Rank 

10.  Computer  Familiarity 

'king 

+ 

F inancia 

7.  •  Yrs  Experience 

15.  Logistics 

Methods 

+ 

+  Higher  Technique  Familiarity  for  Higher  Category  Response 
-  Lower  Technique  Familiarity  for  Higher  Category  Response 


AGE  GROUP  and  LOGISTICS  didn’t  effect  the  treatment 
effected  techniques  for  the  Use  frequency  variable- 
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For  technique  Use  frequency,  LOGISTICS  did  have  a 
difference  for  Decision  Theory  and  Ranking.  The  very  high 
and  very  low  responsibility  levels  marked  the  Decision 
Theory  low  Use  frequency.  The  midlevels  of  responsibility 
markea  Decision  Theory  higher.  The  higher  the 
responsibility,  the  more  Use  frequency  with  the  Ranking 
technique . 

In  the  PRODUCT  DIVISION  category.  Aerospace  Systems 
Division  (ASD)  measured  lower  than  the  Other  response  in 
Decision  Theory,  Ranking  and  Financial  Methods  for  the 
technique  Use  frequency.  The  Other  response  consisted  of 
managers  assigned  to  test  centers.  Air  Force  major  command 
headquarters.  Ballistic  Missile  Systems  Division,  etc. 

Those  that  identified  theoiselves  as  Program  Managers 
generally  used  Statistical  Analysis,  Linear  Programming  and 
Simulation  less  than  those  that  identified  as  Project 
Managers  or  Other.  The  Other  responses  Included  test 
engineer,  division  chief,  branch  chief,  etc. 

ACQUISITION  PHASE  response  categories  and  technique  Use 
were  related,  the  earlier  the  PHASE,  the  more  often  they 
used  the  techniques. 
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Table  12 

Kruakal -Wal 1 is  Teat  for  Influence  of  Demographic  and 
Logistic  Categories  on  Technique  Uae  Frequency 


9  Question's  Subject 

Observed 
Influence  on  Use 

Statistical  Analysis 

5.  SPO  Type 

-  SPO  +  Other 

6.  Job  Position 

-  Pgm  +  Other/Pjt 

Linear  Programming 

6.  Job  Position 

-  Pgm  +  Other/Pj  t 

10.  Computer  Familiarity 

+ 

1 1 .  Computer  Use 

+ 

Simulation 

5.  SPO  Type 

-  SPO  +  Other 

6,  Job  Position 

-  Pjt/Pgm  +  Other 

Decision  Theory 

1.  Product  Division 

-  ASD  +  Other 

7,  *  Yrs  Experience 

; 

15.  Logistics 

-High/Low  +  Midlevels 

Ranking 

1.  Product  Division 

-  ASD  +  Other 

7.  ♦  Yrs  Experience 

+ 

9.  Acquisition  Phase 

- 

15.  Logistics 

+ 

i 

Networking 

3.  Academic  Degree 

+ 

Financial  Methods 

1.  Product  Division 

-  ASD  +  Other 

7.  *  Yrs  Experience 

+ 

9.  Acquisition  Phase 

- 

1 1 .  Computer  Use 

+  Higher  Technique  Use  for  Higher  Category  Response 
-  Lower  Technique  Use  for  Higher  Category  Response 


AGE  GROUP  and  LOGISTICS  didn’t  effect  the  treatment 
effected  techniques  for  the  compu te r i za t i on  Benefit 


fia 


var i ab 1 e . 


AQE  GROUP  did  have  a  difference  for  technique 
computerization  Benefit  in  Project  Selection.  Those 
respondents  under  thirty  and  over  forty-five  marked  the 
Benefit  higher  than  the  other  AGE  GROUP  respondents.  This 
meant  that  despite  the  differences  between  the  control  and 
treatment  group’s  distributions  for  AGE  GROUP,  it  had  no 
measurable  effect  on  the  treatment  effected  techniques. 

For  technique  computerization  Benefit,  LOGISTICS  did 
have  a  difference  for  Linear  Programming,  Project  Selection 
and  Ranking.  The  very  high  and  very  low  responsibility 
level  generally  marked  Linear  Programming  low  Benefit.  The 
midlevels  of  responsibility  marked  Linear  Programming 
higher.  The  higher  the  responsibility,  tne  more  Benefit 
with  the  Project  Selection  and  Ranking  technique. 

In  the  PRODUCT  DIVISION  category,  ASD  measured  lower 
than  the  OTHER  response  in  Ranking  for  the  technique 
computerization  Benefit. 

In  the  FIELD  OF  STUDY  category,  the  OTHER  response  was 
lower  than  the  ENGINEERING,  SCIENCE  AND/OR  MATH  and  BUSINESS 
AND/OR  MANAGEMENT  responses  for  the  technique 
computerization  Benefit  of  Linear  Programming  and  Project 
Selection.  The  OTHER  response  included  Education,  Public 
Administration,  English,  History,  etc. 
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Table  13 


Kruskal -Wal 1 is  Test  for  Influence  of  Demographic  and 
Logistic  Categories  on  Technique  Computerization  Benefit 


•  Question’s  Subject 

Observed  Influence 
on  Bene  fit 

Statistical  Analysis 

2 .  Rank 

+ 

3.  Highest  Degree 

+ 

Linear  Programming 

3.  Field  of  Study 

-  Other  +  E/S/M/B/M 

1 1 .  Computer  Use 

+ 

15.  Logistics 

+ 

Project  Selection  I 

3.  Field  of  Study 

-  Other  +  E/S/M/B/M 

8.  Age  Group 

-  Middle  +  High/Low 

1 5 .  Log i s  t i cs 

+ 

1  Decision  Theory 

6.  Job  Position 

-  Pj t  +  Other/Pgm 

Ranking 

1  Product  Division 

-  ASD  +  Other 

9.  Acquisition  Phase 

- 

15.  Logistics 

+ 

Financial  Methods 

7.  ♦  Yrs  Experience 

- 

1  1 .  Computer  Use 

+ 

+  Higher  Technique  Benefit  for  Higher  Category  Response 
-  Lower  Technique  Benefit  for  Higher  Category  Response 


AGE  GROUP  didn't  effect  any  of  the  techniques  for  the 
computer  program  Availability  variable. 

For  technique  computer  program  Availability,  LOGISTICS 
did  have  a  difference  for  Linear  Programming,  Decision 
Theory  and  Ranking.  The  higher  the  responsibility,  the  more 
use  Opportunity  frequency  with  the  techniques,  LOGISTICS 
didn’t  effet.’t  any  of  the  effected  treatment  techniques. 
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In  the  RANK  category,  the  higher  ranks  marked  the  lower 
responses  in  Linear  Programming  and  Financial  Methods  for 
the  technique  computer  program  Availability. 

In  the  SPO  TYPE  category,  those  that  identified 
themselves  as  working  in  a  BASKET  SPo ,  computer  program 
Availability  of  the  technique  Project  Selection  was  more 
than  the  OTHER  and  SINGLE  SPO  responses. 

AGE  GROUP  and  LOGISTICS  didn't  effect  the  treatment 
effected  techniques  for  the  use  Opportunity  variable. 

For  technique  use  Opportunity,  LOGISTICS  did  have  a 
difference  for  Project  Selection  and  Ranking.  The  higher 
the  responsibility,  the  more  use  Opportunity  wl<^h  the 
Project  Selection  and  Ranking  techniques. 

LOGISTICS  appears  to  have  influenced  the  treatment 
effects  for  Project  Selection  and  Financial  Metliod 
techniques.  Logistic  responsibility  effecting  financial 
matters  in  the  program  office  is  logical.  However,  logistic 
responsibilities  is  not  normally  the  overwhelming  factor  In 
program  office  project  selection  It  is  concluded  that  the 
LOGISTICS  effects  on  Project  Selection  was  coincidental  and 
didn’t  alter  the  treatinent  effect.  The  Financial  Methods 
effect  could  be  attributed  to  logistic  responsibility, 
thereftjre  tiwt  treatment  effect  is  discarded. 

The  demographic  influences  are  listed  in  Appendix  J. 

The  detailed  re.sults  for  LOGISTICS  Influence  are  presented 


in  Appendli^  K 


Table  14 


Kruakal -Wal 1 1 a  Teat  lor  Influence  of 
Demographic  and  I.ogiatic  Categoriea  on 
Technique  Computer  Program  Availability 


»  Question’s  Subject 

Observed  Influence 
on  Availability 

Statistical  Analysis 

1.  Product  Division 

-  ASD  +  Other 

6.  Job  Position 

-  Pgm  +  Other/P j  t 

9.  Acquisition  Phase 

- 

Linear  Programming 

1.  Product  Division 

-  ASD  +  Other 

2 .  Rank 

- 

6.  Job  Position 

-  Pgm  +  Other/Pjt 

9.  Acquisition  Phase 

- 

15.  Logistics 

+ 

Simulation 

6.  Job  Position 

-  Pgm  +  Other/Pjt 

9.  Acquisition  Phase 

- 

Project  Selection 

5.  SPO  Type 

-  Single  +  Other/Bskt 

Decision  Theory 

7.  *  Yrs  Experience 

- 

9.  Acquisition  Phase 

- 

1 5 .  Log i s  t i cs 

+ 

Ranking 

6 .  Job  Pos i t i on 

-  Pgm  +  Other/Pj  t 

9.  Acquisition  Phase 

- 

1 5 .  Log i sties 

+ 

Networking 

7.  ♦  Yrs  Experience 

+ 

Financial  Methods 

2  .  Rank 

- 

7.  »  Yrs  Experience 

- 

1  1  .  Computer  Use 

+ 

+  Higher  Technique  Availability  for  Higher  Category  Response 
-  Lower  Technique  Availability  for  Higher  Category  Response 


72 


Table  15 


Kruskal - Wal 1 is  Test  for  Influence  of  Demographic 
and  Logistic  Categories  on  Technique  Use  Opportunity 


♦  Question’s  Subject 

Observed  Influence 
on  Opportunity 

_ 

Statistical  Analysis 

1.  Product  Division 

-  ASD  +  Other 

5 .  SPO  Type 

-  SPO  +  Other 

6.  Job  Position 

-  Pgm  +  Other/Pjt 

11.  Computer  Use 

+ 

Linear  Programming 

3.  Field  of  Study 

-  Other  +  E/S/M/B/M 

5.  SPO  Type 

-  SPO  +  Other 

11.  Computer  Use 

+ 

Simulation 

1.  Product  Division 

-  ASD  +  Other 

Project  Selection 

1 1 .  Computer  Use 

+ 

15.  Logistics 

+ 

Ranking 

1.  Product  Division 

-  ASD  +  Other 

15 .  Logistics 

+ 

Networking 

3.  Highest  Degree 

+ 

7.  >  Yrs  Experience 

Financial  Methods 

7.  *  Yrs  Experience 

- 

1 1 .  Computer  Use 

+ 

+  Higher  Technique  Opportunity  for  Higher  Category  Response 
-  Lower  Technique  Opportunity  for  Higher  Category  Response 


Tabulat lone .  For  most  of  the  techniques,  75%  of  the 
system  acquisition  program  managers  were  at  least  moderately 
familiar.  However,  only  25%  were  moderately  familiar  with 
Linear  Programming  and  Project  Selection  techniques. 
Approximately  10%  of  the  respondents  use  the  techniques, 
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except  Ranking,  often  or  more.  Only  19X  of  the  managers  use 
the  Ranking  technique  often  or  more.  The  majority  believe 
the  techniques  would  be  useful  to  their  decision  making,  If 
computerized.  Many  of  the  techniques  are  available  on  a 
computer.  Few  managers  report  making  decisions  where  these 
techniques  could  be  used  often. 

The  response  frequencies  for  the  decision  support 
questions  are  in  Appendix  L.  All  of  the  survey  data  la  in 
Included  In  Appendix  M. 

Conclusions 

The  experiment  was  conducted  under  similar  environments 
for  both  the  control  and  treatment  groups.  This  minimizes 
the  effects  of  external  factors  disguising  the  treatment 
effect.  The  treatment  allowed  easier  accomplishment  of  the 
class  work.  The  uncontrolled  group  selection  produced 
similar  group  demographics  between  the  control  and  treatment 
groups,  except  for  the  Age  Group  category.  Age  Group  had  no 
measurable  effect  on  the  technique  variables.  This 
minimized  the  effect  of  the  differences.  The  responsibility 
areas  were  similar  between  the  groups.  The  Logistics  area 
probably  influenced  the  treatment  effect  for  Financial 
Methods  technique  Familiarity.  The  responses  to  the 
after-course  survey  were  istent  for  the  responsibility 

areas.  The  surveys  were  answered  thoroughly  by  the 
respondents . 
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The  treatment  had  a  measurable  effect  on  some 
quantitative  decision  support  techniques  variables.  The 
treatment  effected  Statistical  Analysis,  Linear  Programming 
Simulation  and  Project  Selection  technique  Familiarity.  The 
treatment  effected  Statistical  Analysis,  Linear  Programming 
and  Financial  Methods  technique  Use  frequency.  The 
treatment  effected  Project  Selection  for  computer  progi^am 
Availability.  It  also  effected  Financial  Methods  for 
computerization  Benefit,  computer  program  Availability  and 
use  Opportunity.  The  conclusions  from  these  tests  and 
recommendations  for  future  research  are  In  chapter  V. 
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V. 


Concluslona  and  Beconunendatlona 


Thla  chapter  preaenta  a  brief  aununary  of  the  theala 
objective  and  the  methodology  uaed  to  achieve  that 
objective.  Concluaiona  are  drawn  from  information  provided 
through  the  methodology  which  will  directly  anawer  the 
reaearch  queatlona.  Thla  chapter  dlacuaaea  overall  research 
concluaiona  and  Impllcatlona  related  to  the  teaching  of 
aystema  acquisition  program  manager  decision  support 
techniques.  It  concludes  with  recommendations  for  future 
research  efforts. 

Summary  of  the  Reaearch 

A  quantitative  decision  support  technique,  decision 
tree  analysis,  was  taught  in  the  Intermediate  Program 
Management  (SYS-400)  Professional  Continuing  Education  (PCE) 
course  at  Wright-Patteraon  AFB,  OH.  Before  this  research 
began,  the  classes  were  doing  the  decision  tree  calculations 
without  the  aid  of  a  computer.  This  study  focused  on 
whether  modifying  the  SYS-400  curriculum  to  teach  the  use  of 
PC  baaed  decision  tree  computations  to  program  managers 
would  effect  the  frequency  with  which  they  uaed  quantitative 
techniques  in  addressing  actual  program  and  project 
problems . 

Three  months  after  course  completion,  an  after-course 
survey  was  sent  to  each  course  graduate.  The  after-course 
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survey  measured  the  graduate’s  familiarity  with  and  use 
frequency  of  several  quantitative  decision  support 
techniques.  In  addition,  technique  computerization 
benefits,  technique  computer  program  availability  and  use 
opportunity  were  gathered.  Demographics  and  responsibility 
areas  were  collected  to  identify  possible  effects,  other 
than  the  treatment  effect,  obscuring  the  analysis. 

The  survey  gathered  information  on  several  different 
quantitative  techniques  in  an  attempt  to  measure  the  effect 
of  the  classroom  Instruction.  This  study  did  not  try  to 
discover  if  other  teaching  techniques  or  other  situations 
would  alter  the  frequency  of  quantitative  technique  use. 

Conclusions  to  the  Research  Questions 

1.  How  often  do  system  acquisition  program  managers  use 
quantitative  decision  support  techniques,  with  or  without 
the  aid  of  a  computer? 

Very  little  literature  was  found.  Koble  presented  some 
information  on  the  use  of  statistical  analysis  techniques; 
other  techniques  were  not  reported.  The  literature  focused 
on  what  the  manager  accomplished,  not  what  tools  they  used 
to  achieve  their  goals. 

From  this  study,  the  majority  of  the  system  acquisition 
program  managers  were  moderately  familiar  with  most 
techniques.  Very  few  ( lOX)  of  the  managers  use  the 
techniques  often.  The  majority  believe  the  techniques  would 
be  useful  to  their  decision  making  if  computerized,  which 
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many  are.  Few  managers  report  decision  making  opportunities 
that  would  allow  the  use  of  such  techniques. 

2.  What  tools  do  managers  need  to  problem  solve?  What  do 
they  want  to  use  for  help? 

The  active  program  manager  is  always  balancing  the 
various  demands  of  schedule,  cost,  performance, 
maintainability  and  numerous  other  criteria.  The  manager’s 
staff  helps  by  providing  current  information,  vital  to  the 
program  manager.  Therefore,  the  manager  will  establish  an 
organization  that  provides  the  most  current  information. 

The  active  manager  should  use  habitually  reflective  thinking 
to  decipher  that  information.  Habitually  reflective 
thinking  is  aided  by  current  information,  planning  and 
examining  the  alternatives  through  the  decisional  views  of 
economic  rational,  satisficing,  organizational  procedures, 
political  views  and  individual  differences,  all  of  which 
effect  the  active  program  manager’s  choice  of  solution. 

A  personal  computer  is  now  available  to  assist  the 
manager  in  the  utilization  of  this  flood  of  information. 
Decision  support  systems  (DSS)  have  been  developed  to  help 
the  manager  make  more  informed  decisions  providing  various 
options.  It  is  critical  that  the  DSS  not  take  much  time  to 
learn,  develop  or  update  since  the  manager’s  normal  schedule 
of  constant  interruptions  will  not  allow  for  this. 

This  research  has  demonstrated  that  a  properly  designed 
DSS  can  be  effective  in  providing  current  information  to  the 
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active  manager,  but  only  if  DSS  operates  within  the 

manager’s  decisional  views  and  time  constraints. 

3.  Does  a  three  montn  after-course  survey  of  the  treatment 
groups  provide  reliable  results  that  can  be  compared  to  the 
three  month  after  course  surveys  of  the  control  groups? 

The  after-course  survey  did  detect  differences  between 
the  control  and  treatment  groups.  Both  groups’  demographics 
were  similar  with  two  exceptions:  the  treatment  group’s 
lower  median  Age  Group  and  its  higher  responsibility  level 
in  the  Logistics  area. 

The  experiment  was  conducted  under  £  milar  environments 
for  both  the  control  and  treatment  groups  which  minimized 
the  external  factors  from  disguising  the  treatment  effect. 
The  treatment  allowed  easier  accomplishment  of  the  class 
work.  The  uncontrolled  group  selection  produced  similar 
group  demographics  between  the  control  and  treatment  groups,- 
except  for  the  Age  Group  category.  Age  Group  had  no 
measurable  effect  on  the  technique  variables,  minimizing  the 
effects  of  the  control  and  treatment  group’s  Age  Group 
distribution  difference.  The  responsibility  areas  were 
similar  between  the  groups,  except  for  the  Logistics  area 
The  Logistics  area  influenced  only  the  treatment  effect  for 
Financial  Methods  technique  familiarity.  This  treatment 
effect  was  discarded. 

The  responses  to  the  after-course  survey  were 
consistent  for  the  responsibility  areas.  The  surveys  were 
answered  thoroughly  by  the  respondents. 
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The  treatment  had  a  measurable  effect  on  some 
quantitative  decision  support  technique  variables.  The 
analysis  included  the  test,  Wilcoxon  Rank  Sum  test  and 
the  Student's  t  test.  Errors  are  introduced  in  the  test 
by  collapsing  categories  to  achieve  the  minimum  acceptable 
expected  cell  value.  Therefore,  the  X^  test  results  were 
used  as  a  guide  and  as  a  verification  of  the  other  test 
assumptions.  The  Student’s  t  test  is  considered  more 
accurate  in  detecting  actual  differences  between  samples 
than  the  Wilcoxon  Rank  Sum  test.  The  rank  sum  test  results 
were  used  to  verify  the  Student’s  t  text  assumptions. 

Because  the  Student’s  t  test  generally  agreed  with  the 
nonparametr Ic  test  results,  the  results  were  used  to 
determine  effect  of  the  treatment. 

The  treatment  increased  the  technique  familiarity  of 
Statistical  Analysis,  Linear  Programming  Simulation  and 
Project  Selection.  The  treatment  increased  Statistical 
Analysis,  Linear  Programming  and  Financial  Methods  technique 
use  frequency.  The  treatment  increased  Project  Selection 
for  computer  program  availability.  It  also  increased 
Financial  Methods  for  computerization  benefit,  computer 
program  availability  and  use  opportunity.  Table  16  presents 
these  effects. 

The  three  month  after-course  survey  did  allow  enough 
time  for  the  treatment  effects  to  appear. 
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Table  16 

Observed  Treatment  Effects  to  the 
Quantitative  Decision  Support  Technique  Questions 


Category 

Technique  Question  Area 

Familiar 

Use 

Benefit 

Available 

Opportun.’  ty 

Statistical  Analysis 

X 

X 

Linear  Programming 

X 

X 

Simulation 

X 

■ 

Project  Selection 

X 

■ 

mi 

X 

Banking 

Hetworking 

Financial  Methods 

X 

X 

X 

X 

X  means  a  Difference  between  the  Control  and  Treatment  Groups 


4.  What  demographic  characteristics  effect  the  frequency 
of  quantitative  decision  support  technique  use? 

The  Age  Group  and  Logistics  categories  didn’t  effect 
the  treatment  effected  techniques  for  the  use  frequency 
variable . 

Logistics  did  have  a  difference  for  Decision  Theory  and 
Ranking.  The  very  high  and  very  low  responsibility  levels 
marked  the  Decision  Theory  low  use  frequency.  The  midlevels 
of  responsibility  marked  Decision  Theory  higher.  The  higher 
the  responsibility,  the  more  use  frequency  with  the  Ranking 
technique . 
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In  the  product  division  category.  Aerospace  Systems 
Division  (ASD)  measured  lower  than  the  other  response  in 
Decision  Theory,  Ranking  and  Financial  Methods  for  the 
technique  use  frequency.  The  other  response  consisted  of 
managers  assigned  to  test  centers.  Air  Force  major  command 
headquarters.  Ballistic  Missile  Systems  Division,  etc.  This 
could  mean  that  ASD’s  leadership  doesn't  emphasize  decision 
support  use  as  much  as  the  other  offices  that  participated 
in  the  survey.  It  might  mean  that  the  ASD  manager’s  staff 
does  the  analysis  and  presents  the  results  to  the  manager 
for  the  decision.  This  would  isolate  the  manager  from 
actual  technique  use  and  still  use  the  information  to  make 
better  decisions. 

Those  that  identified  themselves  as  program  managers 
generally  used  Statistical  Analysis,  Linear  Programming  and 
Simulation  less  than  those  that  identified  as  project 
managers  or  oth  The  other  responses  included  test 

engineer,  division  chief,  branch  chief,  etc.  The  S'ir.o.ger  ’  s 
staff  might  provide  the  information  from  the  decision 
support  technique  analysis.  Again,  this  would  isolate  the 
manager  from  actual  technique  use  and  still  use  the 
Information  to  make  better  decisions. 

Acquisition  phase  response  categories  and  technique  use 
were  related,  the  earlier  the  phase,  the  more  often  they 
used  the  techniques.  This  might  occur  because  programs  in 
early  phases  f  the  cycle  have  more  cost  effective 
modification  options.  Those  program  managers  whose  progranis 
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are  in  the  later  stages  might  be  reminded  about  the 
information  available  from  decision  support  techniques. 

Those  that  Identified  themselves  as  working  in  a  SPO 
used  the  techniques  leas  than  the  other  category,  Air  Force 
major  command  headquarters,  laboratories,  test 
organizations,  etc.  The  cause  of  this  difference  is 
probably  similar  to  the  causes  listed  for  the  differences 
for  ASD. 

Use  is  directly  related  with  education  level.  Those 
with  more  formal  education,  doctoral  degrees,  use  these 
techniques  more  often  than  those  with  less  education.  As 
the  level  of  education  decreases,  the  use  decreases.  The 
Field  of  Study  category  had  no  effect  on  the  treatment. 

This  might  mean  the  business  and  management  fields  have 
increased  the  use  of  computers  in  their  curriculum.  This 
would  have  eliminated  a  difference  from  the  science, 
engineering  and  math  curriculums. 

Computer  experience  and  use  were  directly  related  to 
technique  use.  The  more  the  experience  and  use,  the  more 
technique  use  there  was.  This  could  be  related  to  the 
computerization  of  decision  support  techniques.  Since  the 
technique  is  available  on  a  computer  and  the  manager  uses 
that  technique,  therefore  the  manager  is  more  experienced 
with  it. 

The  demographic  characteristics  that  show  an  increase 
in  decision  support  technique  use  allow  upper  management  to 
concentrate  their  efforts  on  the  areas  that  need  attention. 
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ASD  leaders  may  or’y  need  to  emphasize  to  their  system 
program  managers  that  quantitative  decision  support 
techniques  provide  important  Information  to  the  decision 
maker.  Program  managers  could  be  reminded  that  the 
responsibility  of  the  decision  is  theirs  and  a  DSS  can 
assist  in  analyzing  their  options.  These  results  show  the 
value  of  graduate  education  and  computer  knowledge.  Both 
areas  should  continue  to  receive  emphasis  from  the  upper 
management . 

5.  The  Defense  System  Management  College  has  identified 
several  decision  support  techniques  that  are  useful  to 
program  managers.  How  often  io  program  managers  use 
decision  support  techniques? 

For  most  of  tie  techniques,  the  majority  of  the  system 
acquisition  program  .-Banagers  were  at  least  moderately 
familiar.  Very  few  of  the  managers  use  the  techniques 
often.  The  majority  believe  the  techniques  would  be  useful 
to  their  decision  making,  if  computerized.  Few  managers 
report  frequent  opportunities  of  making  decisions  where 
these  techniques  could  be  used.  Many  of  the  techniques  are 
available  on  a  computer. 

Conclusions  f rom  the  Analysis 

The  Intermediate  Program  Management  (SYS-400)  students 
were  able  to  devote  more  time  to  learning  the  use  of 
decision  tree  analysis  information  and  not  concentrate  on 
rigorous  computations.  This  study  showed  that  program 
managers  still  do  not  use  qu.*  titative  decision  support 
techniques  frequently  in  their  program  problem  solving. 
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The  treatment  variables:  technique  familiarity,  use 
frequency,  computerization  benefit,  computer  program 
availability  and  use  opporttini ty ,  are  all  related.  Marager.s 
unfamiliar  with  a  technique  will  not  be  able  to  use  it. 

They  will  not  attempt  to  find  a  computer  program  that 
implements  the  technique,  recognize  opportunities  to  use  the 
technique,  or  even  see  any  potential  benefit.  On  the  other 
hand,  managers  familiar  with  the  basics  of  a  technique  might 
use  it  often  with  the  assistance  of  a  computerized  DSS . 

This  would  mean  a  higher  measurement  of  benefit, 
availability  and  opportunity.  If  the  subject  was  very 
familiar  with  a  technique  but  is  unaware  of  a  DSS  that 
implements  it,  the  benefit  would  be  high  but  the  other 
variables  would  be  low. 

The  data  collected  agreed  with  the  above  possibilities. 
This  allows  the  possibility  that  the  subjects  from  the 

I  treatment  group  observed  the  benefits  from  computerizing  one 

I 

I 

I  technique  and  obtained,  learned  and  used  a  DSS,  even  one 

j  that  wasn’t  used  in  the  SYS-400  class. 

The  techniques  used  for  the  in-class  exercise  were 
decision  tree  and  network  analysis  techniques.  The 
treatment  involved  only  the  decision  tree  analysis.  Neither 
technique  was  effected  by  the  treatment.  The  hypothesized 
reasons  for  this  are:  (1)  the  available  computer  programs 
are  not  satisfactory  to  the  user  (2)  three  months  was  not 
enough  time  to  obtain,  learn  and  use  computerized  DSSs  (3) 
enough  program  problems  that  would  benefit  from  decision 
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tree  or  network  analyses  didn’t  occur  within  the  three  month 
period  and  (4)  the  subject  saw  the  benefit  for  the  decision 
tree  technique  and  searched  for.  obtained  and  used  a  DSS  for 
another . 

Some  of  the  effected  techniques  showed  an  increase  In 
technique  familiarity  but  not  an  Increase  in  technique  use. 
This  could  stem  from  two  hypothesized  reasons;  (1)  a 
technique  appeared  to  be  beneficial,  therefore,  the  manager 
learned  something  about  it  and  there  wasn’t  enough  time  to 
use  the  newly  learned  technique  often  enough  to  alter  their 
response  on  use  frequency  and  (2)  despite  control  and 
treatment  group  similarities,  the  treatment  group  was  more 
familiar  with  the  technique  but  used  it  as  frequently  as  the 
control  group. 

Financial  Methods  showed  an  increase  across  almost  all 
of  the  technique  variables.  This  could  be  attributed  to: 

(1)  despite  control  and  treatment  group  similarities,  the 
treatment  group  was  as  familiar  with  the  technique  but  used 
it  more  frequently  than  the  control  group  and  therefore  knew 
of  the  benefits,  availability  and  opportunities  and  (2)  the 
combined  effect  of  younger  program  managers  and  more 
responsibility  in  the  Logistics  area  might  mean  they  use 
Financial  Methods  more  frequently  than  the  other  managers. 
The  junior  managers  might  be  more  familiar  with  Financial 
Methods  than  other  techniques  and  rely  on  it  heavily.  This 
agrees  with  the  economic  rational  decisional  view  of 
decision  making.  An  external  factor  effect,  the  emphasis 
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from  Congress  and  other  upper  management  on  controlling 
program  costs,  has  been  continuous  and  would  have  effected 
both  groups.  It  is  unlikely  that  the  all  of  the  Increases 
variables  could  be  attributed  to  the  treatment.  The  more 
probable  reason  is  the  treatment  group  was  more  aware  and 
used  Financial  Methods  more  than  the  control  group. 

The  two  day  treatment  Increased  the  use  of  quantitative 
decision  support  techniques.  The  computer  program  was 
designed  to  interact  within  the  program  manager’s  hectic 
pace  and  still  provide  meaningful  current  information.  It 
is  doubtful,  but  possible,  that  before  the  treatment  began, 
the  treatment  group  was  more  familiar  with  and  used  more 
Statistical  Analysis  and  Linear  Programming  techniques. 

Teaching  a  decision  support  technique  with  a 
computerized  DSS  does  appear  to  increase  quantitative 
decision  support  technique  familiarity  and  use.  No  matter 
how  beneficial  a  technique  is,  if  it  is  difficult  to  use  and 
implement,  then  those  skills  will  just  gather  dust. 

System  acquisition  midlevel  program  managers  should 
attempt  to  obtain  DSSs  that  complement  their  own  decisional 
views.  The  more  tools  they  have  available,  the  better 
informed  they  will  be.  This  habitually  reflective  thinking 
could  assist  in  preventing  program  problems  or  minimizing 
the  impact. 

In  order  for  a  computerized  DSS  to  help  the  manager,  it 
should  complement  the  manager’s  decisional  views  and  fit 
into  the  manager’s  hectic  schedule.  It  should  provide  very 
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current  information,  take  little  time  to  use  and  provide 


uncertain,  ad  hoc,  external  information. 

The  goal  of  a  decision  support  system  is  to  take 
necessary,  available  data  and  transform  it  into  a  form  that 
is  meaningful  to  the  manager.  The  manager  is  sometimes 
overwhelmed  with  reams  of  printouts  that  are  not  useful  and 
often  confusing.  If  the  form  isn't  informative,  the 
decision  support  system  will  not  be  used. 

Project  managers  have  expressed  a  need  for  certain 
decision  support  techniques  to  be  computerized.  Computers 
should  be  used  to  assemble  data,  synthesize  it  into 
information  and  then  correlate  and  distribute  the 
information . 

A  manager  can  make  a  better  decision  with  more 
information.  A  good  decision  support  system  makes 
information  available  quickly  and  easily.  Decision  support 
systems  help  managers  learn  different  techniques  for  problem 
solving.  However,  if  the  computer  programs  are  difficult  to 
learn  or  does  not  satisfy  the  office’s  goals,  many  of  those 
gains  are  lost.  Dangerous  computerized  decision  errors  must 
be  prevented  in  order  for  managers  to  arr-ent  the  decision 
support  systems.  Computer  hardware  compatib  ' - uy  is  also 
vital . 

Impl ications 

The  system  acquisition  manager  supervisors  should  feel 
confident  that  courses  using  properly  selected  DSSs  will 
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increase  the  use  of  quantitative  decision  technique  use  in 
the  program  office.  The  additional  informational  from  DSSs 
will  allow  the  program  manager  to  chose  between 
alternatives . 

Upper  management  should  motivate  program  managers  to 
explain  tne  rational  behind  some  of  their  major  program 
decisions.  The  DSS  will  provide  the  program  managers  more 
information  to  explain  their  reasons.  This  supervisor 
attention  should  increase  DSS  use. 

Despite  the  fact  this  research  is  not  conclusive, 
future  instructors  of  quantitative  decision  support 
techniques  should  realize  the  benefits  of  teaching  with  the 
DSS  as  a  learning  aid.  It  appears  to  increase  DSS  use  after 
the  student  returns  to  their  work  place.  A  DSS  selected  for 
the  course  doesn’t  have  to  be  the  best.  However,  it  must 
adhere  to  the  guidelines  outlined  above  to  be  effective. 

The  Air  Force  needs  to  utilize  as  many  tools  as  possible  to 
make  more  informed  decisions  that  effect  the  nation’s 
interests . 

Recommondat ions  for  Future  Research 

1.  Identify  a  set  of  program  manager  demographic 
characteristics  that  could  identify  a  group  of  managers  for 
which  this  teaching  technique  might  be  most  effective.  The 
demographic  information  could  be  used  to  determine  if  a 
specific  set  of  criteria  could  be  found  tc  identify  the 
group (s)  of  program  managers  that  are  not  affected  by  the 
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ins  fcr\io  oion  treatment  and  the  groups  that  are  affected  the 
most.  If  a  set  of  predictors  can  be  found,  they  could  be 
used  to  identify  the  type  of  individual  that  is  most  likely 
to  utilize  quantitative  techniques  in  the  SPO.  Those 
predictors  could  then  be  validated  with  another  experiment. 

This  type  of  information  could  be  used  to  develop  the 
type  of  course  curriculum  that  the  program  manager  needs. 

It  could  identify  which  portions  of  the  total  program 
management  population  would  benefit  the  most  from  a  certain 
level  of  instruction.  There  are  three  hypothesized  levels 
of  instruction  on  quantitative  techniques:  a  one  or  two  day 
seminar;  a  two  week  PCE  course;  or  several  months  of  intense 
instruction  such  as  a  graduate  program.  No  single  level  of 
instruction  is  best  for  the  entire  population.  A  regression 
model  could  possibly  be  used  to  separate  the  program  manager 
population  into  categories.  Each  category  could  be  matched 
to  a  level  of  instruction  for  the  best  results. 

The  accurate  identification  of  the  population 
categories  would  allow  upper  management  to  minimize  the 
costs  of  Instruction. 

2.  Repeat  the  after-course  survey  to  the  same  control  and 
treatment  groups  at  a  longer  interval  than  three  months. 

This  would  allow  the  managers  more  time  to  obtain,  learn  and 
use  the  software  that  is  capable  of  supporting  the  types  of 
decisions  they  need  to  make.  The  disadvantages  are  the 
possibility  of  more  external  variables  effecting  the 
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experiment,  the  subject's  commander  may  change,  more  PCE 
classes,  etc. 

3.  Repeat  the  experiment,  but  for  a  period  of  instruction 
that  is  longer  than  two  class  days.  This  would  test  the 
effectiveness  of  different  teaching  techniques.  The  two  day 
course  was  effective  but  the  use  frequency  was  still  low.  A 
longer  course  would  test  for  a  greater  treatment  effect. 
Pretest  and  posttest  results  should  be  used  for  the 
comparisons.  This  is  more  rigorous  and  external  variables 
are  better  accounted  for. 

4.  Repeat  the  survey  but  randomly  select  program  managers 
from  the  entire  system  acquisition  population.  This  won't 
test  the  effectiveness  of  a  teaching  technique  and 
methodology,  but  it  will  inform  the  researcher  how  often 
program  managers,  not  just  Air  Force  system  acquisition 
midlevel  managers,  use  quantitative  decision  support 
techniques  in  their  program  problem  solving. 
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Appendix  A: 


Decision  Tree  Computer  Ciude  for  SYS-400 


Decision  Tree  Computer  Guide 
for  SYS-400 

’’’his  guide  IS  to  help  the  student  become  familiar  with  using  the  ISV 
personal  compurer  program  Supertree  by  Strategic  Decisions  Group,  CA . 
copyrighted.  This  guide  is  intended  for  academic  use  only  during  the  SYS-400 
class.  AFIT  is  not  recommend i ng  this  program  to  any  student.  The  program  is 
intended  to  aid  in  the  student  understanding  decision  tree  theory  and  practices. 

You  will  be  provided  a  laptop  computer  with  the  necessary  program  already 
installed.  BE  CAREFCL.  They  are  fragile 

Open  the  lid  and  adjust  the  screen.  Plug  in  the  computer  to  any  wall 
cutlet  and  plug  the  other  end  inta  the  small  female  plug  on  the  right  edge  of 
the  back  of  the  computer.  It  is  inarked  "DC".  Turn  on  the  computer,  the  switch 
is  on  the  right  side  of  the  computer  towards  the  back. 

After  the  computer  starts  ‘.’p ,  you  will  see  a  menu.  On  the  right  side  o' 
the  menu  you  will  see  'FS  SYS-^00'.  Press  the  'F2'  key  at  the  top  of  your 
keyboard  Cn  the  left  side  of  the  menu  you  will  see  ‘1  ju.nertrce'.  Select 
Supertree  by  pressing  the  "Enter"  key  or  the  number  "1"  near  the  top  of  the 
keyboard . 

When  Supertree  starts  you  will  see  this  screen:  (screens  are  separated  from 
the  text  by  shaded  lines) 


Structure  Evaluate  Analyze  Utility  Programs  Files  Quit 
Input,  Show,  Delete,  Renumber,  Treename,  Clear 
- STODcHT  EDITION - 


SGPERTREE,  7ERSI0H  5.25 
c  STRATEGIC  DECISIOKS  GROUP,  1985 


Esc  to  discard  Input  or  move  to  higher  menu 
PgUp  to  view  previous  screens 
Ctrl  PrtSc  to  use  printer 


These  commands  are  important.  Take  the  time  to  find  these  keys  and  the 
arrow  keys.  If  a  command  lists  two  keys,  bold  down  the  first  then  press  the 
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second.  The  first  row  of  words  are  the  gienu  choices.  The  second  row  is  the 
menu  available  from  the  highlighted  choice  or  a  short  explanation  of  the 
selection.  To  select  from  the  menu,  highlight  the  desired  choice  by  moving  the 
highlight  bar  with  the  right  and  left  arrow  keys.  Or  you  can  press  the  first 
letter  of  the  desired  choice.  The  highlight  bar  Is  displayed  in  this  text  as 
under  1 1 nes , 

Press  the  right  arrow  key  three  times  to  ‘Otility'  and  tho.i  .ss  'Enter' 
or  just  press  'O'.  Select  'Configuration'  and  the  following  screen  will  appea." 
Everything  should  be  setup  for  you  already.  You  may  want  to  change  the 
'Decision  criterion;'  to  'MAXIMIZE'  or  'MINIMIZE'  depending  if  the  model  is  used 
to  maximize  profits  or  minimize  costs. 


CONFIGOP.ATION 

Dse  FI  to  input  screen  data;  Esc  to  discard  screen  data. 

- studek:  edition - 


Decision  criterion; 

Maximize  or  minimize  expected  va 1 ue/cer ta i n  equivalent?  MINIMIZE 
Numerical  formats  (field  width,  decimal  places); 

Format  for  values’  8  0 
Format  for  probabilities?  i  2 
Interval  at  which  running  total  of  models  run  is  shown?  25 
Dse  line  graphics?  YTS 

Default  drive  for  Supertree  (files,  models,  worksheets,  etc.)?  C 

Drive  on  which  to  find  system  for  Lotus,  Symphony,  IFPS,  FCS  or  Igultiplan?  C 

Drive  for  configuration  file,  SDPEET.CNF?  C 

Color  combination  used  on  screen  (7=B&W)?  7 

Which  port  is  printer  attached  to?  LPTl 

Maximum  column  width  of  printer:  144 

Printer  setup  message:  15 


The  printer  setup  message  sets  the  printer  to  condensed  print  (17cpi).  If 
you  want  to  change  to  10  cpi ,  delete  the  15  and  change  the  maximum  column  width 
to  80 . 

After  you  have  made  all  of  the  changes  you  want,  press  'FI*.  If  you  made 
mistakes  and  want  to  start  over,  press  'Esc* .  You  can  move  up  and  down  the 
input  screen  with  the  arrow  keys.  As  you  enter  information  in  any  of  the  input 
screens,  the  old  information  is  erased  in  that  field  when  you  press  your  first 
key.  If  the  field  has  only  a  few  choices,  only  the  first  one  or  two  letters  are 
required.  The  program  will  supply  the  rest.  For  example,  if  you  wanted  to 
change  'Decision  criterion;'  from  'MINIMIZE'  to  'MA.XIMIZE'  (as  you  would  need  to 
do  for  the  Decision  Tree  Bidding  example),  you  would  only  have  to  type  'MA*  and 
press  'Enter".  Supertree  will  fill  In  the  rest.  "Y"  is  enough  for  an  answer  to 
a  yes/no  question.  Supertree  capitalizes  all  of  your  inputs  automatically.  If 
you  can  not  get  capitalized  letters,  press  the  ' Num  Lock'  key.  If  this  doesn't 
work,  ask  for  assistance. 
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Be  sure  to  periodically  save  your  work 


Structure  Evaluate  Analyze  Otility  Programs  Files  Quit 
Savefile,  Kecallfile,  Wscopy,  Directory 
- ^ — - STUDEK'T  EDITIOK - 


S a V e ( 1 1 e  Recallfile  Wscopy  Directory 

Save  structure  and  values  of  a  tree  to  a  file 

- STtrDEK”i'  EDITION 

Name  under  which  to  save  file. 


Savefile  Reca 1 1 f i 1 e  Wscopy  Directory 

Recall  structure  and  values  of  a  tree  from  a  file 

- - - - - - STUDENT  EDITION- 

Name  of  drive.  C 
Files: 


BIDEX 

Name  of  file  to  be  recalled: 


After  you  have  drawn  out  your  tree  manually,  you  will  want  to  enter  it 
into  Supertree.  To  create  a  new  tree,  press  'Esc'  until  the  main  menu  apoeans. 
Select  'Structure'.  Select  'Input'.  The  following  screen  will  appear.  answer 
tae  screen  prompts.  On  your  manually  drawn  tree  number  the  different  nodes 

Normally,  nodes  are  numbered  starting  from  '1'  and  increase  to  the  right  This 

IS  not  mandatory  for  this  program  but  it  will  help  you  keep  track  of  which 
branch  goes  where.  For  this  progr<im.  nodes  are  only  different  when  the  prev  ous 

decision  types  and  states  of  nature  are  not  the  same. 

The  example  shown  in  class  is  repeated  here  to  demonstrate  some  features 
of  the  program.  (See  the  Decision  Tree  Bidding  Example  figure  on  page  4) 
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DECISION  TREE 

BIDDING  EXAMPLE 


WIN  3 


/  N  c  ci  6  ^ 


V  / 


BID  ON  3 


-  1  0 


\  5/  /■ 

'V  /o. 


\  0 . 6 


Vv  1  f  A 


LOSE  3 


BID  ON  A 


hjO  BID  B 


\_  ^  /  V'l/ !  ‘ ; 


ON  3 


i 


LOSE  A 


\  LOSE  5 


NO  BID  B 


WIN  B 


BID  ON  B 


NO  BID  A 


C.6 


\  0.4 
\ 

\ 


LOSE  B 


NO  BID 


FIGURES  ARE  VALUES  IN  THOUSANDS 
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Enter  a  unique  number  fp'  Humber  of  node  (positive) 

Node  type  can  be  OUNCE,  DECISION  or  ENDPOINT  (C,D  or  E) 

Number  of  branches  is  the  number  of  cho ioes/poss I bi 1 i t i e s  available  from 
that  node 

This  first  screen  shows  the  error  message  when  you  try  to  enter  an 
incorrect  node  type. 


Input  Show  Delete  Beniimber  Treename  Clear 
Add  or  change  node  Input 

POSSIBLE  RESPONSES  ARE:  DECISION  CHANCE  ENDPOINT 

Number  o f  node :  9 
Node  type:  S 


Input  Show  Delete  Renumber  Treename  Clear 
Add  or  change  node  input 

- STUDENT  EDITION - 

Enter  a  descriptive  name  for  this  tree.  BIDDING  TREE 
Number  of  node  :  ]_ 

Node  type:  DECISION 
Number  of  branches:  2 


Enter  a  short,  unique  and  descriptive  name,  such  a.,  BIDA. 


DECISION/CHANCE  NODE  ENTRY 

Use  El  to  input  screen  data;  Esc  to  discard  screen  data. 
NODE  NUMBER; i 

TYPE : DECISION  OUTCOMES  DEPEND  ON  NODES; 

NUMBER  OF  BRANCHES ■ 2 
NODE  NAME:  _ _ 

SUCCESSOR 

OUTCOME  REWARD  NODE 
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Dae  only  letters  and  numbers  for  the  node  name.  The  program  will  tell  you 
if  you  used  illegal  characters. 


DECISION/CHAUCE  MODE  ENTRY 

Use  El  to  input  screen  data;  Esc  to  discard  screen  data. 

FIRST  CHARACTER  kfUST  BE  A  LETTER.  REST  LETTERS  OR  NUMBERS,  NO  BLANV.S - 

MODE  ITJWBER;  1 

TYPE.DECISIOH  OUTCOkfES  DEPEND  ON  NODES; 

NUNCBER  OF  BRANCHES:  2 
NODE  NAME: BID  A 


Enter  the  results  of  the  decision  or  state  of  nature  under  'CUTCOME 
Dse  the  arrow  keys  to  edit  the  inputs  When  finished,  press  'FI' . 


DECISION/CHANCE  NODE  ENTRY 

Use  FI  to  input  screen  data;  Esc  to  discard  screen  data. 

- STDDENT  EDITION - 

NODE  NUMBER: 1 

TYPE. DECISION  OUTCOMES  DEPEND  ON  NODES: 

NUMBER  OF  BRANCHES : 2 
NODE  NAME  3  I  DA 

SUCCESSOR 

ODTCOME  REWARD  NODE 

2  — 

3 
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Select  'Input'  to  enter  the  next  node. 


DECI SI  OH/CHANCE  NODE  ENTRY 

Use  FI  to  Input  screen  data;  Esc  to  discard  screen  data. 

- STUDENT  EDITION - — - 

NODE  NUk(BER;2 

TYPE  CHANCE  OUTCOMES  DEPEND  ON  NODES: 

NUMBER  OF  BRANCHES  ■  2  PROBABILITIES  DEPEND  ON  NODES'. 

NODE  NAMEWINA 

SUCCESSOR 

PROBABILITY  OUTCOME  REWARD  NODE 


DECISION/CHANCE  MODE  ENTRY 

Use  FI  to  input  screen  data;  Esc  to  discard  screen  data 

- STUDENT  EDITION - 

NODE  NUMBER; 4 

TYPE. DECISION  OUTCOMES  DEPEND  ON  NODES: 

NUMBER  OF  BRANCHES : 2 
NODE  NAMEiBIDAB 

SUCCESSOR 

OUTCOME  reward  NODE 
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\i  the  outcome  or  probabilities  are  dependent  on  a  chain  ol  previous 
node(s),  enter  those  node  numbers  in  the  spaces  on  the  top  right  ol  the  entry 
screen.  Thus,  when  we  get  to  node  5,  both  the  probabilities  and  the  outcomes 
depends  on  what  happens  at  node  2. 


DECISIOH/CBANCE  NODE  ENTRY 

Use  El  to  input  screen  data;  Esc  to  discard  screen  data. 


NODE  NUMBER; 5 


TYPE .CHANCE 

NUMBER  OF  BRANCHES  2 

NODE  NAME:BIDAWINB 

SELECT  TO 
ENTER/REVIEW 
PROBABILITIES 


OUTCOMES  DEPEND  ON  NODES; 2 
PROBABILITIES  DEPEND  ON  N0DES;2 

SELECT  TO  SUCCESSOR 

ENTEH/RBVIEW  REWARD  NODE 

OUTCOMES 

- 0 - 0  — 

- 0 - 0  — 


Select  the  probabilities  lield.  The  next  screen  appears.  On  the  left  side 
is  the  output  of  the  nodes  that  this  node  depends.  On  the  right  is  where  you 
enter/review  the  specific  probabilities  or  outputs.  To  step  through  each 
possible  combination,  use  the  'Tab'  Icey . 


Use  Tab/Shift  Tab  To  Scroll  Through  Conditioning  Nodes. 

Use  FI  to  Return  to  Node  Entry  Screen;  Shift  PrtSc  to  Print  Screen. 
- STUDENT  EDITION - 


WINA 


PROBABILITY  FOR  BIDAWINB 


0 


r' 


8 


99 


Use  Tab/Shift  Tab  To  Scroll  Through  Conditioning  Nodes. 

Use  FI  to  Return  to  Node  Entry  Screen,  Shift  PrtSc  to  Print  Screen. 
- STUDENT  EDITIOM^ — - - 


WIKA 


PROBABILITY  FOR  BIDAWINB 


50 

>0 


Next,  enter  the  correct  outcomes  for  each  situation. 


Use  Tab/Shift  Tab  To  Scroll  Through  Conditioning  Nodes. 

Use  FI  to  Return  to  Node  Entry  Screen;  Shift  PrtSc  to  Print  Screen. 
- STUDENT  EDITION - 


WIKA  OUTCOME  FOR  BIDAWINB 


0 


Use  Tab/Shift  Tab  To  Scroll  Through  Conditioning  Nodes. 

Use  FI  to  Return  to  Node  Entry  Screen;  Shift  PrtSc  to  Print  Screen, 
- STUDENT  EDITION - 


WINA  OUTCOME  FOB  BIDAWINB 


50 

>0 


r 


9 


100 


For  the  Bidding  Tree  example,  when  MODE  5  is  complete,  this  is  what  your 
Inputs  should  look  like. 


DECISIOK/CRAUCE  MODE  EJ.TRY 

Dse  FI  to  input  screen  data;  Esc  to  discard  screen  data. 

- STUDEHT  EDITIOM - 

HODE  HUMBER; 5 


TYTErCHAHCE 

HUMBER  OF  BRAHCHES ; 2 

HODE  NAMEiBIDAWIHB 

SELECT  TO 
EHTER/BEVIEW 
PROBABILITIES 


L 


OUTCOMES  DEPEND  OH  H0DES:2 
PROBABILITIES  DEPEND  OH  NODES: 2 

SELECT  TO  SUCCESSOR 

EHTER/REVIEW  REWARD  HODE 

OUTCOMES 

- 0 - 6  — 

- 0 - 6  — 


To  review  what  you  have  entered,  sel'ct  'Show'.  The  following  screen  will 
appear.  If  it  scrolls  off  the  screen,  press  the  'PgUp',  'PgDn'  or  the  arrow 
keys  to  see  what  you  want.  The  key  to  the  output  under  the  heading  ’STRUCTDTIE' 
is:  NODE  *,  HODE  TYPE,  and  SUCCESSOR  NODES. 


Input  Show  Delete  Renumber  Treename  Clear 
Show  the  structure  entered  for  the  tree 

- STUDEHT  EDITION 

Tree  name;  BIDDING  TREE 


STBDCTURE 

NAMES 

OUTCOMES 

PROBABILITIES 

1  D  2 

3 

BIDA 

-10  0 

2  C  4 

4 

WIHA 

50  0 

.5  .5 

4  D  5 

7 

BIDAB 

-10  0 

5  C  6 

6 

BIDAWIHB 

Depends  on  2 

Depends  on  2 

VINA 

OUTCOME,  NODE 

5 

C-50  — 

20 

0 

L- 0 - 

50 

0 

WIHA 

PROBABILITY, 

NODE  5 

C-50  — 

0.4 

0.6 

1-0 - 

0.7 

0.3 

>>  TREE  CONTAINS  SUCCESSORS  WHICH  ARE  NOT  DEFINED  AS  NODES 


I 

To  correct  the  Inforiaation  in  a  node,  select  ’Input'  to  get  the  node  entry 
screen.  Use  the  arrow  keys  to  select  the  desired  field  and  enter  the  correct 
Information.  There  Is  no  means  of  editing  the  field’s  entry.  The  entire  field 
must  be  retyped.  If  node  4's  1st  branch  outcome  was  entered  as  ’lO’,  the 
correcting  entry  would  be  ’-10’.  The  minus  sign  can't  be  just  inserted. 

At  the  end  of  the  tree  we  have  ' EWDPOINT ’ s .  Supertree  can  calculate  the 
values  of  the  endpoints  In  our  example  as  follows.  To  enter  endpoints  that  can 
have  variable  values,  a  formula  must  be  entered  The  formula  must  start  with  ’Bs’ 
to  let  the  program  know  a  formula,  not  a  variable  name  of  this  node,  follows. 

The  formula  is  then  entered,  parenthesizes  are  optional.  The  following  screen 
shows  summing  the  values  of  the  nodes  that  led  to  this  endpoint.  The  node  names 
are  the  variable  names .  The  node  numbers  are  then  entered  on  the  next  line. 


EMDPOINT  ENTRY 

Dse  FI  to  input  screen  data;  Esc  to  discard  screen  data. 

- STUDENT  EDITION - 

NODE  NUMBER; 6 


TYPE:  ENDPOINT 

Name/value  of  endpoint: 

B* (BIDAtWINA+BIDABtBIDAWINB) 

Nodes  upon  which  endpoint  depends: 
12  4  5 


Be  sure  to  periodically  save  your  work. 


Structure  Evaluate  Analyze  Utility  Programs  Files  Quit 
Savefile,  Becallfile.  Wscopy,  Directory 
- STUDENT  EDITION - 


Savef lie  Eecallfile  Wscopy  Directory 

Save  structure  and  values  of  a  tree  to  a  file 

- STUDENT  EDITION 

Name  under  which  to  save  file:C:BIDEX 


1 1 
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Here  is  the  structure  of  the  complete  tree. 


Tree  name:  BIDDING  TREE 


STROCTDRE 

NAMES 

OUTCOMES 

PROBABILITIES 

1  D  2  3 

BIDA 

-  10  0 

2  C  A  4 

WINA 

50  0 

.5  .5 

3  D  8  9 

NOABIDB 

-10  0 

4  D  5  7 

BIDAB 

-10  0 

5  C  6  6 

BIDAWINB 

Depends 

on 

2 

Depends  on  2 

6  E 

BS  (BIDA-t  WINA»BIDA& 
B+BIDAWINB) 

Depends 

on 

1 

2 

4  5 

7  E 

BS  {BIDAfWIHA^BIDAi 

B) 

Depend.s 

on 

1 

2 

4 

8  C  10  10 

NOAWINB 

50  0 

.6  .4 

9  E 

B* (BIDA*NOABIDB) 

Depends 

on 

1 

3 

10  E 

Bf  (B!DA  +  NOABIDB«-N& 

Depends 

on 

1 

3 

8 

OAWIfTB) 


WINA 

OUTCOME.  NODE  5 

C-50  — 

20  0 

i— 0 - 

50  0 

WINA 

PEOBABILITY,  NODE  5 

C-50  — 

0.4  0.6 

L_o - 

0.7  0.3 

Press  'Esc'  to  backup  to  the  main  menu.  Select  'Evaluate'.  You  need  to 
evaluate  the  tree  anytime  the  node  inputs  are  changed . 


Structure  Evaluate  Analyze  Otilizy  Programs  Files  Quiz 
Calculate  branch  values  using  the  endpoint  models/values 

- STUDENT  EDITION - 

Supertree  will  check  input  compatibility  and  will  then  evaluate  endpoints; 
this  may  take  some  time. 

Do  you  wish  to  continue  with  this  evaluation?  YES 
>>  TOTAL  OF  9  MODELS  BUN 

>>  ENDPOINT  VALUES  ASSIGNED  BIDDING  TREE 
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Select  ’Analyze'  to  get  to  the  ’Analyze'  submenu. 


Structure  Evaluate  Analyze  Utility  Programs  Files  Quit 

Hollback,  Display.  Plot.  List.  Sensitivity.  Atti tude- to-Sisk .  Trace 

- STUDENT  EDITION - ^ - 


Select  ’Display’  to  show  the  tree  drawn. 


Rollback  Display  Plot  List  Sensitivity  Trace  Attitude-to-Risk 
Show  portions  of  evaluated  tree 

- ^STUDENT  EDITION - 


If  you  want  tbe  order  of  the  nodes  to  remain  the  same,  just  press  ’Enter'. 


display 

Use  FI  to  input  screen  data;  Esc  to  discard  screen  data. 

- STUDENT  EDITION - 

Present  Order  of  Nodes: 

1  2  4  5  3  8 
New  Order  of  Nod’s: 

1  2  4  5  3  8 

First  Node:  1 
Last  Node;  8 

Single  or  double  spacing?  SINGLE 
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The  output  to  the  screen  will  look  like  the  screen  below.  If  you  want  to 
print  the  tree  also,  press  'Ctrl  PrtSc".  ‘PBIMTEB'  will  appear  on  the  right 
edge  of  the  screen  just  below  the  menus.  Use  the  arrow  (cursor)  keys  to  view 
parts  of  the  tree  that  are  off  the  screen,  if  any.  To  print  the  entire  screen, 
press  "Shift  PrtSc". 


Use  Cursor/Shift  Cursor  keys  to  scroll  off  screen;  Esc  to  return  to  main  menu. 

- STODEKT  EDITION - PEIHTER 

BIDA  EXP  VAL  PROS  WINA  EXP  7AL  BIDAB  EXP  VAL  PROS  BIDAWIHB  EXP  VAL 


r>-10 

-i 

— 0  — 


.50-50 


.  50-0 


40 


r 


10- 


u>o- 


154 


>-10- 


38-C-O  .  40-20 - 50  — 

>—0.60-0 - 30  — 

40 - 

15-C -0.70-50 - 30  — 

>—0.30-0 - 20  — 

10 - 


■20 


Press  "Shift  -->"  to  see  the  rest  of  the  tree. 


Use  Cursor/Shift  Cursor  keys  to  scroll  off  screen;  Esc  to  return  to  main  Ea,iU. 

- STUDENT  EDITinv-  - PRINTER 

BIDAB  EXP  VAL  PROS  BIDAWIKB  EXT  7aL  NOABIDB  EXP  VAL  PROB  NOAWINB  EXP  VAL 


.  1  n- 


-6 

L)0- 


4 


>-10- 


>-0- 


-38-C-O.  40-20- 
>— 0.60-0  — 
-40 - 


-1  5-C -0.70-50- 
>-0.30-0  — 
■10 - 


-50- 

-30- 


-30- 

-20- 


r>-10- 


>— 0- 


-20-C -0.60-50- 
>-0.40-0  — 
— 0 - 


-40 

■10 


14 


105 


This  is  what  the  printed  output  looks  like  with  80  columns. 


BIDA  EXP  VAL  PBOB  WIMA  EXP  VAL  BIDAB  EXP  VAL  PROB  BIDAWINB  EXP  VAL 


r->-10- 

D 


0- 


u. 


50-50- 


50-0- 


-20- 


-40 


-T 


10- 


<->0- 


-15h5^ 


>-10- 


>— 0- 


-30-K:-O  ,40-20- 
‘—0  60-0  — 
-40 - 


-!  5-C -0.70-50- 
•—0.30-0  — 
10 - 


-50 - 

-30 - 


-30- 

-20- 


NOABIDB  EXP  VAL  PBOB  NOAWINB  EXP  VAL 


r>-10- 


•-<3- 


-2.0-C -0.60-50- 
•—0.40-0  — 
— 0 - 


>)  Tree  Drawn 
>>  Expected  Value;  28 


-40- 

-10- 


SIDOrf/G  TREE 


loe 


Change  the  order  of  nodes,  If  necessary,  to  display  only  the  parts  of  the 
tree  you  want.  Enter  the  nodes  wanted  In  the  desired  order,  press  'Enter'  and 
the  progran  will  fill  in  the  rest  of  the  nodes.  Then  enter  the  first  node  to  be 
displayed  and  the  last  node  to  be  displayed. 


Present  Order  of  Nodes: 

1  2  4  5  3  8 
New  Order  of  Nodes: 

1  3  8  2  4  5 

First  Node:  1 
Last  Node:  8 

Single  or  double  spacing?  SINGLE 


BIDA  EXP  VAL  NOABIDB 
>  -  10 - 28 - 


■— 0- 


-20-^ 
*— 0 


)-10- 


EXP  VAL  pros  HOAWINB  exp  VAL 


■20-C -0.60-50 - 4  0- 

‘-0.40-0 - 10- 

-0 - 


>>  Tree  Dravm  BIDDING  TREE 

>>  Expected  Value:  28 


If  you  want  to  analyze  the  tree's  sensitivity  to  changes  in  a  probability 
distribution  at  a  node,  select  'Sensitivity'.  Sensitivity  to  risk  tolerance  is 
also  available. 


Rollback  Display  Plot  List  Sens i ti vi ty  Trace  Alti tude- to-Risk 
Perform  sensitivity  analysis  on  probabilities  or  risk  tolerance 
- STUDENT  EDITION - 


Probabi 1 i ty  Bisk 

Perform  sensitivity  to  variations  in  probability  at  a  node 
- STUDENT  EDITION - 
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Prooaoiiiiy  Risk 

Perform  sensitivity  to  variation  in  the  risk  tolerance 
- — - STUDEy?  EDITIOH - 


PROBABILITY  SENSITIVITY 

Use  FI  to  enter  screen  data;  Esc  to  discard  screen  data. 

- - - STUDENT  EDITION - 

Present  Order  of  Nodes. 

1  2  4  5  3  8 
New  Order  of  Nodes: 

1  2  4  5  3  8 

For  which  node  do  you  want  the  sensitivity  to  probability?  2 
Small  or  large  sized  plot?  SMALL 


Because  the  sensitivity  requires  multiple  tree  evaluations, 
it  may  take  some  time  to  run. 

Smallest  and  largest  values  encountered  are:  15  40 
Lower  and  upper  limits  for  vertical  axis-  10  45 


* 
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If  the  first  node  Ie  a  decision,  the  sensitivity  will  be  shown  for  all  branches. 
The  vertical  axis  la  the  expected  value. 

The  horizontal  axis  is  the  probability  of  the  first  branch  of  node  2; 
the  last  branch  of  node  2  has  the  remainder  of  the  probability. 


>>  Probability  Sensitivity 


BIDDING  THEE 


45 


38 


24 


17  h  1 


1  0  I - ‘ i - J - 1 - ■ - i - i - 1 - 1 - 1 - 1 

,0  .1  .2  .3  .4  .5  .6  ,7  .8  .9  I.O 


PgUp'  to  view  previous  screens 


Probability  Risk 

Perform  sensitivity  to  variation  in  tbe  risk  tolerance 
- STUDENT  EDITION - 


Present  Order  of  Nodes; 

1  2  4  5  3  8 
New  Order  of  Nodes; 

1  2  4  5  3  8 

Vary  risk  tolerance  from  infinity  down  to'’  2 
Small  or  large  sired  plot7  SMALL 
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Because  the  sensitivity  requires  multiple  tree  evaluations, 
the  program  may  take  some  time  to  run. 

Smallest  and  largest  certain  equivalents  are  -9  128 
Lower  and  upper  limits  to  be  used  for  plot:  -10  30 

If  the  first  node  is  a  decision,  the  sensitivity  is  shown  for  all  branches. 
The  vertical  axis  is  the  certain  equivalent. 

The  horicunlal  axis  is  the  risk  tolerarce; 

the  scale  is  linear  in  1/risk  tolerance. 


'i  Risk  Sensitivity 


BIDDING  THEE 


Rollback  Display  Plot  List  Sensitivity  Trace  Attitude-to-Risk 
Set  the  risk  tolerance  to  be  used  In  tree  analysis 
- STODEMT  EDITION - 

Do  you  wish  to  use  the  exponential  utility  function?  YES 
Input  the  value  for  the  Bisk  Tolerance:  _ 


IS 
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Be  sure  to  periodically  save  your  work. 


Structure  Evaluate  Analyse  Utility  Programs  Fi 1 r s  Quit 
Saveiile.  aecall/lle,  Wscopy,  Directory 
- - - STUDENT  EDITION - 


To  exit  the  program,  press  'Esc'  to  get  to  the  main  menu.  Select  'Quit'. 


Structure  Evaluate  Analyze  Utility  Programs  Files  Quit 
Quit  Supertree 

— - - - STUDENT  EDITION - 


No  Tes 

Return  to  fcUin  Supertree  Menu 

- STUDENT  EDITION 


No  Yes 

Quit  Supertree  Now  (Have  You  Saved  Your  Work"?) 
- STUDENT  EDITION 


Quick  Reference  of  Important  Commands 

'Esc'  to  discard  input  or  move  to  higher  menu 

'PgUp'  to  view  previous  screens 

"Ctrl  PrtSc"  to  use  printer  (on  and  off) 

'Shift  PrtSc'  to  print  entire  screen 

Use  Bigbt/Left  arrow  keys  to  highlight  menu  choices  and  press  'Enter'  to  select 
OB  Select  menu  choices  by  pressing  the  first  letter  of  that  choice 
All  formulas  must  start  with  'BS'  when  the  tree  la  not  linked  to  other  programs 
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Appendix  B:  SYS-400  Af ter -Course  Survey 


DEPARTMENT  OF  THE  AIR  FORCE 
air  UMIV£HSITY 

air  force  institute  of  technology 

WRIGHT-PATTERSON  AIR  FORCE  BASE  OH  45433 


Kt'.T 10 
aPTk  Of 

SUS.tCI 


LSY  (Capt  Donahue.  AV  785-3355) 
SYS' A 00  After  Course  Survey 


13  FEB  89 


'o  Irta  j  John  Q.  Doe  III 


1.  A  survey  of  recent  SYS'400  graduates 
gather  information  about  decision  making, 
this  survey  will  help  us  improve  the  tool 
to  effectively  manage  their  programs. 

2.  Please  take  the  t i ae  to  complete  the  attached  survey  and 
return  it  by  19  JUN  89  The  inforcration  collected  will  serve  as 
the  basis  for  an  AFIT  graduate  student  thesis  on  the  us“  of 
computers  as  an  aid  in  program  management  decision  making.  All 
participants  in  this  survey  will  remain  anonymous  and 
participation  is  voluntary, 

3.  Thank  you  for  your  support  in  this  -.•esearcb  effort.  Should 
you  require  further  information,  please  contact  me  at 

AV  785-3355,  If  you  are  interested  in  receiving  a  sum-mary  of  the 
results  of  this  research,  please  write  to; 

AFIT/LSG 

ATTK :  Capt  Mark  Donahue 

Wri gh t -Pa t terson  AF3 .  OH  45433-65S3 


is  being  taken  to 
Your  responses  to 
s  available  for  managers 


EDWARD  J.  KOw-LaRD.  Lt  Col.  USAF 
SYS-400  Course  Director 

AFIT  School  of  Syste ms  and  Logistics 


STRENGTH  THROUGH  kNOWLEDGE 


SURVEY  FOR  SYS-400 


Do  NOT  put  your  name  or  SSN  on  the  answer  sheet  or  questionnaire. 
This  inlornistion  is  strictly  anonymous.  No  1  dent  1 1  Icat  1  on  required. 

For  each  question,  select  the  answer  most  appropriate  lor  you. 
Please  mark  all  your  responses  on  the  answer  sheet  with  a  No.  2  pencil 
and  completely  blacken  the  circle.  Include  BOTH  the  answer  sheet  and 
the  questionnaire  in  the  return  envelope. 


1.  For  which  product  division  do  you  work'? 

1.  ELECTRONIC  SYSTEM.S  DIVISION 

2.  AERONAUTICAL  SYSTEMS  DIVISION 

3.  SPACE  DIVISION 

4.  OTHER  (Specify) _ 


What  is  your  rank? 


1  .  CAPTAIN  OR  BELOW 

2.  MAJOR 

3.  LIEUTENANT  COLONEL  OR  ABOVE 

4.  GM-12,  GS-12  OR  BELOW 

5.  QM-13,  GS-13  OR  ABOVE 


3.  What  is  the  highest  academic  degree  you  hold'? 


1 .  BACHELORS  DEGREE 

2.  MASTERS  DEGREE 

3.  DOCTORATE  DEGREE 

4 .  NONE  OF  THE  ABOVE 


4.  In  what  field  is/are  your  degree(s)  (mark  all  that  apply)? 

1.  ENGINEERING 

2.  SCIENCE  AND/OR  MATH 

3.  BUSINESS  AND/OR  MANAGEMENT 

4.  OTHER  (Specify) _ 


6  . 


How  would  you  classify  the  SPO  in  which  you  work? 

1.  BASKET  SPO  (more  than  one  weapon  system) 

2.  SINGLE  SYSTEM  SPO 

3.  OTHER  (Specify) _ 

I  am  a : 


1.  SPO  DIRECTOR  (Chief  of  an  entire  SPO) 

2.  PROGRAM  MANAGER  (Single  Manager  of  a  complete  system  - 

works  (or  the  SPO  director) 

3.  PROJECT  MANAGER  (Manager  of  a  program's  system  subc'^mponent  - 

works  for  a  Program  manager) 

4.  OTHER  (Specify) _ _ 


113 


7.  How  much  cumulative  experience  do  you  have  as  a  government  program 
or  project  manager  (in  whole  years,  at  one  half,  round  up)? 

1 .  HOME 

2  .  1 

3  .  2-3 

4  .  4-6 

5 .  MORE  THAM  6 

a.  Please  mark  your  age  group? 

1 .  UMDER  30  YEARS  OLD 

2.  30-35  YEARS  OLD 

3  36-40  YEARS  OLD 

4  41-45  YEARS  OLD 

5.  OVER  45  YEARS  OLD 

9.  What  is  the  current  p’lase  of  your  program(s>?  (Mark  more  than  one 
response  if  you  Lave  programs  in  different  phases.) 

1.  COKCEPT  EXPLORATION 

2.  DEMONSTRATION/VALIDATION 
FULL  SCALE  DEVELOPMENT 

4.  PRODUCTION  AND  DEPLOYMENT 

10.  How  familiar  are  you  with  using  a  personal  computer  or  a  mainframe 

computer  ? 

1 .  NONE 

2.  SLIGHTLY 

3 .  MODERATELY 

4 .  VERY 

5.  COMPLETELY 

11.  How  often  do  you  personally  use  a  computer  for  your  work? 

1 .  NEVER 

2.  LESS  THAN  1  HOUR  A  DAY 

3.  1  to  2  HOURS  A  DAY 

4.  3  to  4  HOURS  A  DAY 

5.  MORE  THAN  4  HOURS  A  DAY 
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For  the  following  questions,  aiarlc  the  appropriate  answer  based  on 
the  following  scale  of  answers: 

1  2  3  4  5 

NE 


I  make/have  made  program/pro j ect  decisions  related  to: 


12  . 

cost  and  budget 

1 

2 

3 

4 

5 

13  . 

schedu 1 e 

1 

2 

3 

4 

5 

14  . 

Lec;..iicai  perfor  ma  nee 

1 

2 

3 

4 

b 

15  . 

logistics 

1 

2 

3 

4 

5 

16  . 

administration  of  program  personnel 

1 

2 

3 

4 

5 

17  . 

establishing  spares  levels  and  deciding 
other  spares  associated  issues 

1 

2 

3 

4 

5 

18  . 

establishing  prototype  or  production 
schedules 

1 

2 

3 

4 

5 

19  . 

selection  of  SPO  personnel 

1 

2 

3 

4 

5 

20  . 

Program  Objective  Memorandum  inputs 

1 

2 

3 

4 

5 

2  1  . 

establishing  the  contents  of  the 

Integrated  Logistics  Support  Plan 

1 

2 

3 

4 

5 

22. 

approval  of  Engineering 

Change  Proposals 

1 

2 

3 

4 

5 

23  . 

selection  of  reliability  criteria 

1 

2 

3 

4 

5 

24  . 

designation  of  job  responsibilities 
for  other  SPO  personnel 

1 

2 

3 

4 

5 

25. 

setting  of  specification  requirements 

1 

2 

3 

4 

5 

26  . 

establishing  major  milestone  dates 

I 

2 

3 

4 

5 

27  . 

supplying  gov't  furnished  equipment 
versus  contracting  for  the  equipment 

1 

2 

3 

4 

5 

28  . 

setting  dates  for  ma jor  revie ws 

1 

2 

3 

4 

5 

29  . 

redirecting  contractor  technical  effort 

1 

2 

3 

4 

5 

30. 

types  of  procurement  data  purchased 

1 

2 

3 

4 

5 

31  . 

system  cost  estimates 

1 

2 

3 

4 

5 
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A  list  of  quantitative  technique  categories,  which  may  have  been 
valuable  to  you  as  a  program  or  project  manager,  is  on  the  next  several 
pages.  Answer  the  following  five  questions  for  each  category. 

QUESTIONS 

Fami 1 i ar  -  Indicate  how  familiar  you  are  with  the  one  or  more  of  the 
techniques  in  each  category 

Use  -  For  each  category,  indicate  how  often  you  (not  others)  now  use  or 
have  used  one  or  more  of  these  techniques  for  program  management. 

Dene  fit  -  In  each  category,  how  often  could  one  or  more  of  these 
techniques  be  helpful  to  you  in  making  decisions  In  your  program? 

Aval  1 ab 1 e  -  For  all  of  the  techniques  in  each  category,  what  proportion 
are  available  with  the  aid  of  a  personal  computer  or  mainframe  computer? 


Oppor  tun i ty  -  Disregarding  time  limitations,  how  often  do  you  make 
decisions  where  one  or  more  of  these  techniques  could  have  been 
used? 


Statistical  Analysis  -  the  collecti 
of  data  according  to  -.-.ell  defined  s 
sampling,  averages,  regression,  and 

on,  organization  and  inte 
tatistical  procedures.  I 

time  series  analysis. 

r  p  r  e  t  a  t  1  0  n 
nc 1 udes 

7 

2 

1 

3 

4 

5 

32 

Farm  1  i  ar 

WOKE 

1 

SLIGHTLY 

i 

MODERATELY 

i 

VERY 

1 

COMPLETELY 

53 

Use 

KEVEE 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

o  4 

E  e  7.  e  f  1  t 

NEVE?. 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

35 

Available 

NONE 

SOfvfE 

HALF 

MOST 

ALL 

w  C? 

Oopor  tun i tv 

NEVE? 

SELDOM 

OCCASIONALLY 

OFTEN 

always 

Linear  programming  and  other  math  programming  methods  -  a  gr 
techniques  for  optimizing  the  use  of  limited  resources.  Inc 
integer  progra  mm  irig,  nonlinear  prog:*  a  min  mg,  and  dyriamic  pr  g 

oup  of 

1  uc  e  £ 

1  simmi  n  g  . 

1 

1 

2 

1 

3 

4 

5 

37 

F  a  m.  1  1  1  a  r 

1 

NONE 

1 

SLIGHTLY 

I 

MODERATELY 

VERY 

COMPLETELY 

3  S 

Use 

N  E  V  E  H 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

39 

B  e  fi  e  f  it 

NEVE? 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

40 

Aval lable 

NONE 

SOME 

HALF 

MOST 

ALL 

4  1 

Oopor  tun i t v 

NEVE? 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

lie 


Simulation  -  a  computerized  ma the ma t i ca 1  -  1 o g i ca 1  model  of  a  real  system 
or  situation  Experimentation  with  the  siir.ulation  aids  in  decisions 
surrounding  the  system  or  situation.  Includes  reliability  analysis. 


1 

2 

3 

[ 

4 

5 

42 

Far.  1  1  lar 

N(!»K'£ 

SLIGHTLY 

MODERATELY 

veAy 

1 

COMPLETELY 

43 

Use 

NEVER 

SELDOM 

OCCASIO.NALLY 

OFTEN 

ALWAYS 

45 

Benefit 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

46 

NONE 

SOME 

HALF 

MO  S  T 

ALL 

4  7 

Oozortunitv 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

Project  selection  models 
the  selection  of  Research 
y.o  1 1 1  e  y  -  i.'e  wt  on  model.  Die 
t  e  c  h  r.  1  c  u  e  s  . 

-  this  inc 
and  Devel 
man  mode  1  , 

■ludes  models  developed  for 
opment  projecis.  Includes 

goal  orcgramming  and  Delp 

aiding  in 
l'n& 

K  < 

1 

2 

3 

4 

5 

48 

F  a.mi  liar 

NONE 

1 

SLIGHTLY 

1 

MODERATELY 

VE^lY 

1 

COMPLETELY 

49 

Use 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

50 

Benefit 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

AL^'.’AYS 

5  1 

Ava i 1 ab 1 e 

NONE 

SOME 

HAL? 

MOST 

ALL 

52 

Oooor  tun  1 ty 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

Decision  theory  -  a  systematic  technique  for  quantifying  complex- 
decisions  and  analyzing  alternatives.  Includes  the  use  of  decision 
trees,  influence  diagrams  cr  payoff  tables.  P.lsc  includes  utility 
theory,  risk  analysis  and  game  theory. 


1 

1 

2 

1 

3 

I 

4 

I 

5 

53  . 

F  ami  1 1 ar 

1 

NONE 

1 

SLIGHTLY 

MODsitATELY 

1 

VERY 

1 

COMPLETELY 

54  . 

Use 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

55  . 

Benefit 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

56  . 

Available 

NONE 

SOME 

HALF 

MOST 

ALL 

57  . 

OoDor  tun i tv 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 
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Ranking  -  a  technique  of  putting  factors  related  to  decisions  in  a 
ranked  order  of  importance  based  on  a  survey  of  knowledgeable 


1 nd i V 1 dua  1  s  - 

Includes 

index  numbers 

.  checklists  and 

1  A  n  a  1  y  t  1 

cal 

Hierarchy  Process. 

1 

2 

3 

4 

5 

58 

Fami 1 iar 

NONE 

SLIGHTLY 

MODERATELY 

veI.v 

CO.MPLETELY 

5  9 

Use 

K  E  V  E  R 

SELDOM 

OCCASIONALLY 

often 

ALWAYS 

£  0 

Ser.cf  *  t 

.VEVZ  A 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

6  1 

Aval  1 ab 1 e 

NONE 

SOME 

HALF 

.MOST 

ALL 

62 

Oppo  r  tun  1  tv 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

Ke  two  r  k i ng  -  a 

g  r  a  p  h  1  c  a 

i  description 

of  a  problem  or 

■  £ i t  ua  t  i 

0  n 

consisting  of 

discrete 

activities  related  to  time  and  to  each  other. 

Includes  PERT  , 

CPV. ,  Gar. 

1 1  charts  and 

milestone  chart 

•  £ 

2 

3 

4 

5 

63 

Fa mi  1 iar 

j _ 

NONE 

SLIGHTLY 

modeRately 

veRy 

1 

COMPLETELY 

64 

Use 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

65 

Benefit 

NEVE?. 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

65 

Aval lable 

NONE 

SOME 

HALF 

MOST 

ALL 

67 

Ooocr  tun i ty 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

Financial  methods  -  thi 

E  1 nc 1 udes  a 

group  of  financi 

a  1  e va 1 u 

a  t  i  on 

techniques  including  equivalent  uniform  annual  cost, 

present 

value  , 

internal  rate 

of  re  turn 

.  net  present 

value ,  payback 

per  1 od  , 

f inancial 

percent  return 

on  i n ve  s 

tment,  cost-benefit  analysis, 

C/SCSC  , 

??ES  and 

Life  Cycle  Cos 

t . 

1 

I 

2 

3 

4 

5 

68 

Fami liar 

N^NE 

SLIGHTLY 

modeRately 

veRy 

COMPLETELY 

69 

Use 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

70 

Benefit 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

7  1 

Aval lable 

NONE 

SOME 

HALF 

MOST 

ALL 

7  2 

Oppor  tun i t V 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

IIB 


other 


Specify  below  (Leave  answer  sheet  BLANK  if  NOT  used! 


1 

2 

_ _ _ 1 _ 

3 

1 

4 

1 

5 

73 

Farm  1  lar 

NCDK'E 

1 

SLIGHTLY 

MODERATELY 

) 

VERY 

- 1 

COMPLETE 

7  4 

Use 

NEVER 

SELDOM 

0CCA3ICNALLV 

C  FT  EN 

A  L  WA  V  S 

75 

£  e  n  e  f  :  t 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

A.  LV.'AYS 

7  6 

Aval  1  a  b 1 e 

NONE 

SOME 

HALF 

MO  S  T 

ALL 

77 

OsDor  tun i t v 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

Other  -  Specify 

below 

(Leave  answer 

sheet  BLANK  i f 

NOT  used) 

1 

}  ■  -  -  - 

2 

1 

3 

1 

4 

5 

78  . 

Familiar 

1 

NONE 

SLIGHTLY 

MODERATELY 

VERY 

1 

COMPLETE 

7C.  . 

Use 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

eo . 

Benefit 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

3  1  . 

Aval lable 

NONE 

SOME 

HALF 

MOST 

ALL 

32  . 

Oopor  tun i t v 

NEVER 

SELDOM 

OCCASIONALLY 

OFTEN 

ALWAYS 

110 


If  you  have  any  comments  concerning  the  use  of  computers  as  an  aid 
to  decision  making  for  program  managers  or  anything  else,  please  include 
them  in  the  space  below. 

Comments : 


Include  BOTH  this  survey  and  the  answer  sheet  in  the  return  envelope. 
Thank  you  for  your  time  and  cooperation. 
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Appendix  C:  W1 1 coxon  Hank  Sum  Test  Beeul ts  of  Comparison 
between  Control  and  Treatment  Groups  *  Demograph 1 os 


Question 

Wi  1 coxon 
T 

Probabi 1 i ty 

1  . 

Product  Division 

0.149 

0.8816 

2. 

Rank 

1  .  165 

0. 2439 

3. 

Highest  Degree 

0.  160 

0 . 8732 

4  . 

Field  of  Study 

0.790 

0 . 4295 

5  . 

SPO  Type 

0.468 

0.6396 

6  . 

Job  Position 

0.968 

0.3328 

7  . 

Cumulative  Experience 

0.963 

0 . 3355 

8  . 

Age  Group 

1 .958 

0.0502 

9  . 

Acquisition  Phase 

1.340 

0 . 1802 

10. 

Computer  Familiarity 

0.521 

0.6020 

1 1  . 

Computer  Use 

0.591 

0 . 5547 
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Appendix  D:  Test  Beaul ts  of  Compar Ison  between 

Control  and  Treatment  Groups’  Responsibility  Areas 


Main  Area 


Question 

D 

Probabi 1 i ty 

12.  Cost  and  Budget 

1 . 990 

2 

0 . 370 

13.  Schedule 

3.513 

2 

0 . 173 

14.  Technical  Performance 

0.267 

2 

0.875 

15.  Logistics 

4.054 

2 

0. 132 

16.  Admin  of  Pgm  Personnel 

2. 166 

4 

0 . 705 

Specific  Area 


Question 

X=* 

a 

Probabi 1 i ty 

20. 

Cost  and  Budget 

1 .835 

3 

0.607 

27  . 

Cost  and  Budget 

8.448 

4 

0.076 

31  . 

Cost  and  Budget 

7.  169 

3 

0.067 

18  . 

Schedule 

4.006 

3 

0.261 

26  . 

Schedule 

3.228 

2 

0.199 

28. 

Schedule 

3.786 

4 

0.436 

22  . 

Technical  Performance 

1.746 

4 

0 . 782 

25. 

Technical  Performance 

1  .  124 

4 

0.890 

29. 

Technical  Performance 

9.883 

4 

0 . 042 

17. 

Logistics 

3.141 

2 

0 . 20 

21  . 

Logistics 

5.096 

2 

0 . 078 

23  . 

Log i sties 

5.258 

2 

0 . 072 

30  . 

Log i s  t i cs 

10.602 

4 

0-031 

19. 

Admin  of  Pgm  Personnel 

0. 525 

3 

0.913 

24  . 

Admin  of  Pgm  Personnel 

3 . 040 

4 

0.551 
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Appendix  E:  Wi Icoxon  Bank  Sum  Teat  Heaul ta  of  Comparlaon 
between  Control  and  Treatment  Qroupa ’  Reanonalbl 1 1 ty  Areas 


Main  Area 


Quest 1  on 

Wi  Icoxon 
T 

Probabi 1 i ty 

12.  Cost  and  Budget 

1.314 

0. 1807 

13.  Schedule 

0.341 

0.7334 

14.  Technical  Performance 

0.154 

0.0774 

15.  Logistics 

1  .  B98 

0.0577 

16.  Admin  of  Pgm  Personnel 

1.117 

0.2638 

Specific  Area 


Question 

Wi  1 coxon 
T 

Probabi 1 i ty 

20 

Cost  and  Budget 

0.676 

0 . 4992 

11 

Coat  and  Budget 

1.618 

0 . 1057 

31 

Cost  and  Budget 

1 . 564 

0.1177 

10. 

Schedule 

1 . 277 

0.2016 

26 

Schedule 

0.027 

0.9784 

28 

Schedule 

1.655 

0.0979 

22 

Technical  Performance 

0.266 

0.7902 

25 

Technical  Performance 

0.293 

0.7698 

29 

Technical  Performance 

2.  160 

0 . 0307 

17 

Logistics 

1 . 362 

0 . 1731 

21 

Logistics 

2.091 

0.0365 

23 

Log i s  t i cs 

1.591 

0.1116 

30 

Logistics 

2.208 

0 . 0272 

19 

Admin  of  Pgm  Personnel 

0.210 

0.0273 

24 

Admin  of  Pgm  Personnel 

0.458 

0 . 6472 
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Appendix  F:  Teat  Beaul ta  of  Compap 1 aon  between 

Control  and  Treatment  Qroupa  *  Quantitative 
Deciaion  Support  Teohniaue  Reaponaea 


Statiatical  Analyaia  Category 


Ques  1 1  on 

D 

Probabi 1 i ty 

32. 

Fami 1 iar 

7.528 

0.023 

33. 

Use 

5.898 

0.052 

34. 

Benef i t 

2.722 

0 . 256 

35. 

Avai labl e 

7.940 

0.094 

36. 

Oppor tun i ty 

1 . 548 

1 

0.213 

Linear  Programming  and  Other  Math  Programming  Methods 


Ques  t i on 

X^ 

m 

Probabi 1 i ty 

37  . 

Fami 1 iar 

9  107 

3 

0 . 028 

38  . 

Use 

5  606 

2 

0.061 

39. 

Benef i t 

5.020 

2 

0.081 

40  . 

Ava i 1 ab 1 e 

0.941 

2 

0 . 625 

41  . 

Opportun i ty 

1.526 

2 

0.466 

S i mulat i on 


Question 

X=» 

o 

Probabi 1 i ty 

42. 

Fami 1 i ar 

4.961 

2 

0.084 

43. 

Use 

0.959 

2 

0.619 

44  . 

Benef  i  t 

6 . 268 

3 

0 . 099 

45  . 

Ava i 1 ab 1 e 

0.292 

2 

0  864 

46. 

Opportun i ty 

2.824 

2 

0. 244 

Project  Selection 


Quest i on 

X^* 

D 

Probabi 1 i ty 

47  . 

Fami 1 iar 

12.641 

2 

0 . 002 

48  . 

Use 

0.403 

2 

0.818 

49. 

Benef  i  t 

5.227 

2 

0 . 073 

50. 

Avai labl e 

4 . 593 

2 

0.101 

51  . 

Opportun i ty 

1 . 536 

2 

0 . 464 
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Decision  Theory 


Question 

X* 

52  . 

Fami 1 lar 

3.030 

53  . 

Use 

2.334 

54  . 

Bene  f  i  t 

2.800 

55. 

Avai lable 

3.250 

56  . 

Opportuni ty 

1 . 535 

Ranking 

Quest  ion 

mm 

57  . 

Fami 1 i ar 

5 . 606 

58  . 

Use 

2.126 

59. 

Benef  i  t 

2.367 

60  . 

Avai lable 

4.351 

61  . 

Opportun i ty 

3.825 

Probabi 1 i ty 


0.220 
0.311 
0.422 
0.355 
0. 464 


Probabi 1 i ty 


0.231 
0. 547 
0 . 669 
0. 226 
0 . 430 


Networking 


Quest ion 

X=* 

D 

Probabi 1 i ty 

62  . 

Fami 1 iar 

1.635 

2 

0.442 

63  . 

Use 

0.246 

2 

0.084 

64  . 

Benef i t 

3.356 

3 

0. 187 

65. 

Available 

1 . 477 

3 

0.680 

66. 

Opportunity 

0.595 

2 

0.743 

Financial  Methods 


Question 

m 

Probabi 1 i ty 

67  . 

Fami liar 

10.444 

0.015 

68. 

Use 

12.117 

■I 

0.017 

69  . 

Bene  fit 

8 . 567 

D 

0.073 

70  . 

Avai lable 

8.066 

n 

0.089 

71  . 

Opportun i ty 

12.089 

n 

0.017 
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Appendix  Q;  W1 Icoxon  Rank  Sum  Teat  Beaul ta  of  Comoar laon 
between  Control  and  Treatment  Groupa  *  Quantitative 
Declalon  Support  Technlaue  Beaponaea 


Statlatlcal  Analyala 


Question 

Wi  1 coxon 
T 

32. 

Fami 1 iar 

2.544 

33. 

Use 

1.932 

34. 

Benef i t 

1.288 

35. 

Avai lable 

1 .378 

36. 

Opportuni ty 

1.479 

0.0110 
0.0534 
0. 1978 
0. 1604 
0 . 1391 


Linear  Programming  and  Other  Math  Programing  Methods 


Question 

Wi  Icoxon 
T 

Probabi 1 i ty 

37.  Familiar 

2.874 

0.0041 

38.  Use 

1.405 

0. 1601 

39.  Benefit 

1.098 

0.2721 

40.  Available 

0.639 

0.5227 

41.  Opportunity 

0.826 

0.4005 

Simulation 


Question 

Wi  1 coxon 
T 

Probabi 1 i ty 

42 .  Fami 1 iar 

2.416 

0.0157 

43.  Use 

0.027 

0.9788 

44 .  Bene  fit 

0.128 

0.8984 

45.  Available 

0.326 

0.7442 

46.  Opportunity 

0.069 

0.9448 

Project  Selection  Techniques 


Quest i on 

Wi  Icoxon 
T 

Probabi 1 i ty 

47.  Familiar 

2 . 430 

0.0151 

40.  Use 

0.287 

0.7738 

49.  Benefit 

0.906 

0.3651 

50.  Available 

1 . 623 

0. 1046 

51.  Opportunity 

0 . 963 

0 . 3355 
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Declalon  Theory 


Question 

Wi  Icoxon 
T 

Probabi 1 i ty 

52  . 

Fami 1 iar 

1.043 

0 . 2970 

53. 

Use 

0.700 

0.4842 

54. 

Benef i t 

0.049 

0.9611 

55. 

Avai lable 

0.229 

0.8190 

56  . 

Opportunity 

0.372 

0.7095 

_ 1 

Ranking 

Wi  Icoxon 

Question 

T 

Probabi 1 i ty 

57. 

Fami 1 iar 

1 . 203 

0.2291 

58. 

Use 

0.937 

0.3490 

59. 

Benef i t 

0.843 

0.3994 

60. 

Available 

0.810 

0.4179 

61  . 

Opportunity 

0.832 

0.4055 

Networking 

Wi  Icoxon 

Question 

T 

Probabi 1 i ty 

62. 

Fami liar 

0.894 

0.3713 

63. 

Use 

0.245 

0.8066 

64  . 

Bene  fit 

0.809 

0 .4186 

65. 

Avai lable 

0.740 

0.4595 

66  . 

Opportunity 

0.479 

0.6320 

Financial  Methods 


Question 

Wi 1 coxon 
T 

Probabi 1 i ty 

67.  Familiar 

2.717 

0.0066 

68 .  Use 

2.837 

0.0046 

69 .  Benef i t 

2.386 

0.0170 

70.  Available 

2.488 

0.0128 

71.  Opportunity 

2.899 

0.0037 
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Appendix  H;  Student's  t  Test  Besults  of 

Comparison 

between  Control  and  Treatment  Groups’ 

Q 

uantl tati ve 

Declglon  Support  Technique  Beaponaea 


Statlatlcal  Analyaia 


Question 

t 

df 

Probabi 1 i ty 

32.  Familiar 

-2.o32 

02.5 

0.008 

33 .  Use 

-1 . 999 

71 .8 

0.049 

34.  Benefit 

-1.432 

68.  1 

0.157 

35.  Available 

-1 .484 

68.  1 

0. 143 

36.  Opportunity 

-1.575 

71 . 5 

0. 120 

Linear  Programming  and  Other  Math  Programing  Methods 


Question 

t 

df 

Probabi 1 i ty 

37.  Familiar 

-3.084 

66.6 

0.003 

38.  Ose 

-1.678 

63.4 

0.098 

39 .  Benef i t 

- 1 . 063 

71 .8 

0.292 

40.  Available 

-0 . 558 

66 . 1 

0.579 

4 1 .  Opportunity 

-0.726 

70.5 

0.470 

Simulation 


Question 

t 

df 

Probabi 1 i ty 

42.  Familiar 

-2.740 

66.7 

0.008 

43.  Use 

-0.201 

66.6 

0.841 

44 .  Benef i t 

0.049 

72.6 

0.961 

45.  Available 

-0.422 

66.7 

0.675 

46.  Opportunity 

-0.028 

71.8 

0 . 97^ 

Project  Selection  Techniques 


Question 

t 

df 

Probabi 1 i ty 

47.  Familiar 

-1.997 

71.8 

0.050 

48.  Dae 

-0.504 

61.5 

0.616 

49.  Benefit 

-0.664 

70.8 

0.509 

50.  Available 

-1.733 

59.5 

0.088 

51 ,  Opportunity 
_ 1 

-0.969 

66 . 9 

0 . 336 

Decision  Theory 


Question 

t 

df 

Probability 

52.  Familiar 

-1 . 273 

71.8 

0.207 

53.  Use 

-0.581 

71.8 

0.563 

54.  Benefit 

0.038 

71.7 

0.970 

55.  Available 

-0.077 

72.4 

0.939 

56.  Opportunity 

-0.580 

69.0 

0.564 

Ranking 


Question 

t 

df 

Probabi 1 i ty 

57.  Familiar 

-1.337 

72.4 

0. 185 

58.  Use 

-0.984 

67.6 

0.328 

59.  Benefit 

-0.980 

69.0 

0.330 

60.  Available 

-0.768 

67.3 

0.445 

61.  Opportunity 

-0.912 

69. 1 

0 . 365 

Networking 


Question 

t 

df 

Probab i 1 i ty 

62.  Familiar 

-0 . 993 

69.5 

0 . 324 

63.  Use 

0.095 

72.5 

0 . 925 

64.  Benefit 

0.590 

72.9 

0.557 

65.  Available 

0.697 

71.8 

0.488 

66.  Opportunity 

0.347 

71.3 

0.730 

Financial  Methods 


Question 

t 

df 

Probabi 1 i ty 

67.  Familiar 

-3. 164 

68.8 

0.002 

68.  Use 

-3. 161 

69.0 

0 . 002 

69 .  Bene  fit 

-2.614 

70 . 7 

0.011 

70.  Available 

-2 . 632 

69.9 

0.011 

71.  Opportunity 

-3 . 158 

69.3 

0.002 
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Appendix  I;  Means  of  Control  and  Treatment  Groups  * 
Quantitative  Decision  Support  Technique  Responses 


Statistical  Analysis 

COMTHOL  TREATMENT 


N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

32. 

Familiar 

41 

2 . 76 

0.830 

34 

3.38 

1  045 

33. 

Use 

41 

1 . 88 

0.927 

34 

2 . 29 

0.871 

34  . 

Benef 1 t 

41 

2.51 

0.810 

34 

2.79 

0.880 

35. 

Available 

40 

2.45 

1.280 

34 

2.91 

1 . 379 

36. 

Opportunity 

41 

2.39 

0.802 

34 

2  68 

0.7O8 

Linear  Programming  and 

Other  Math  Programing  Methods 

CONTROL 

TREATMENT 

N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

37. 

Fami 1 iar 

41 

2.04 

1 .048 

34 

2.85 

1  .  184 

38  . 

Use 

41 

1 .44 

0.673 

34 

1 . 74 

0.828 

39. 

Bene  fit 

41 

1.93 

0.985 

33 

2.  15 

0.834 

40  . 

Aval lable 

41 

1.93 

1 . 253 

32 

2.09 

1 . 279 

41  . 

Opportunl ty 

41 

1.90 

0.970 

33 

2.06 

0 . 899 

Simulation 

CONTROL 

TREATMENT  I 

N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

42. 

Fami 1 iar 

41 

2.53 

1  ^27 

34 

3.24 

1.156 

43. 

Use 

41 

1 . 95 

0.973 

34 

2.00 

1.101 

44. 

Benef 1 t 

41 

2.51 

1.143 

34 

2 . 50 

1 .022 

45. 

Aval lable 

41 

2 . 29 

1 . 289 

33 

2.42 

1  .S-ro 

46. 

Opportunl ty 

41 

2.32 

1 .035 

34 

2.32 

0 . 976 

Project  Selection  Techniques 


CONTROL  TREATMENT 


N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

47. 

Fami liar 

40 

1 .75 

1  .  104 

34 

2 . 24 

0.987 

48  . 

Use 

40 

1 .35 

0 . 662 

34 

1 . 44 

0.860 

49. 

Benef 1 t 

40 

1 . 65 

0.947 

34 

1.79 

0.914 

50. 

Aval lable 

41 

1 . 39 

0.771 

34 

1 .76 

1 . 046 

51  . 

Opportunl ty 

41 

1.61 

0.891 

34 

1 . 82 

0.999 
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Decision  Theory 


COHTBOL  TREATMEMT 


N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

52. 

Fami liar 

41 

2.98 

0.907 

34 

3.24 

0.855 

53. 

Use 

40 

2.08 

1.023 

34 

2.20 

0.914 

54. 

Benef 1 t 

40 

2 . 45 

1.061 

34 

2.44 

0.960 

55. 

Aval lable 

41 

2 . 24 

1 .220 

34 

2.26 

1  .  109 

56. 

Opportunity 

41 

2.20 

0.928 

34 

2.32 

0 . 976 

Ranking 

CONTROL 

TREATMENT 

N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

57. 

Fami liar 

41 

2.63 

1 . 240 

34 

2.97 

0 . 937 

58. 

Use 

40 

2 . 18 

1 .059 

33 

2.42 

1.091 

59. 

Benef i t 

41 

2 . 49 

1.052 

33 

2.73 

1.039 

60. 

Available 

41 

2. 10 

1.281 

33 

2.33 

1.339 

61  . 

Opportunity 

41 

2.34 

1  .  109 

33 

2.58 

1.091 

Networking 

CONTROL 

TREATMENT 

N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DEV 

62. 

Fami liar 

41 

3.68 

0.850 

34 

3.88 

0.880 

63. 

Dae 

41 

3. 17 

1  .  138 

34 

3. 15 

1 .019 

64  . 

Benefit 

41 

3.34 

1  .  109 

34 

3.21 

0.880 

65  . 

Aval lable 

41 

3.32 

1 . 274 

34 

3. 12 

1 . 200 

66. 

Opportunl ty 

41 

3.15 

1  .  108 

34 

3.06 

1.071 

Financial  Methods 


CONTROL 


TREATMENT 


N 

MEAN 

STD  DEV 

N 

MEAN 

STD  DE' 

67. 

Fami 1 iar 

40 

2.88 

1.017 

34 

3.65 

1 . 070 

68. 

Dae 

39 

2 . 28 

1  .  146 

34 

3. 15 

1  .  184 

69  . 

Benef  i  t 

40 

2 . 63 

1 .213 

34 

3.35 

1  .  178 

70  . 

Aval lable 

39 

2.44 

1.353 

34 

3 . 26 

1 . 333 

71  . 

Opportunity 

39 

2 . 36 

1 . 224 

33 

3.24 

1  .  146 
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Appendix  J:  Demographic  Inf luenoe  on 
Decision  Support  Technique  Responses 


1. PRODUCT  DIVISION 


Kruskal -Wal I Is 


Question  on 

df  =  3 

Faml 1 1 ar 1 ty 

Probabi 1 1 

32. 

Statistical  Analysis 

3.35 

0.3403 

37. 

Linear  Programming 

5.44 

0. 1421 

42  . 

Simulation 

5 . 94 

0.1144 

47. 

Project  Selection 

1.71 

0.6354 

52. 

Decision  Theory 

3.30 

0.3475 

57. 

Ranking 

3.90 

0.2729 

62. 

Networking 

3. 18 

0.3653 

67. 

Financial  Methods 

5.47 

0 . 1402 

I. PRODUCT  DIVISION 

Kruskal-Wal 11s 

Question  on 

df  =  3 

Use 

X= 

Probabi 1 1 

33. 

Statistical  Analysis 

6. 15 

0 . 1047 

38. 

Linear  Programming 

5.31 

0 . 1507 

43. 

Simulation 

5.88 

0  1  177 

48. 

Project  Selection 

2.81 

0.4222 

53. 

Decision  Theory 

8.50 

0.0353 

58. 

Ranking 

6 . 46 

0.0912 

63. 

Networking 

2.25 

0.5221 

68. 

Financial  Methods 

7. 17 

0.0666 

1. PRODUCT  DIVISION 

Kruskal -Wal 1 1  s 

Question  on 

df  =  3 

Bene  fit 

X=> 

Probabi 1 1 

34. 

Statistical  Analysis 

1 . 20 

0.7521 

39. 

Linear  Programming 

3.62 

0.3054 

44. 

Simulation 

5 . 04 

0 . 1689 

49. 

Project  Selection 

5.67 

0. 1287 

54. 

Decision  Theory 

2 . 60 

0 . 4582 

59. 

Ranking 

7 . 17 

0.0667 

64. 

Networking 

1 . 07 

0.7846 

69  . 

Financial  Methods 

6.03 

0. 1099 

ty 


ty 


ty 
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l.PBODDCT  DIVISION 
Question  on 
Aval lability 
3S.  Statistical  Analysis 
40.  Linear  Prograamlng 
45.  Simulation 
50.  Project  Selection 
55.  Decision  Theory 
60.  Banking 
65.  Networking 
70.  Financial  Methods 


Kruakal -Wal 1 is 
df  =  3 


x= 

Probabi 

6.85 

0 . 0770 

6.50 

0.0808 

6.21 

0. 1016 

4.40 

0.2213 

4.80 

0. 1860 

5.77 

0. 1232 

2.32 

0.5082 

4. 19 

0.2416 

l.PBODDCT  DIVISION  Kruskal -Wal 1 i a 


Question  on 

df  =  3 

Opportunity 

Probabi 1 1 ty 

36. 

Statistical  Analysis 

7.22 

0.0651 

41  . 

Linear  Programming 

4.53 

0.2095 

46. 

Simulation 

7.72 

0.0521 

51  . 

Project  Selection 

4.61 

0 . 2025 

56. 

Decision  Theory 

2.68 

0.4437 

61  . 

Banking 

9. 14 

0 . 0275 

66. 

Networking 

2. 18 

0.5351 

71  . 

Financial  Methods 

4 . 05 

0.2558 

2 . BANK 

Kruskal -Wal 11s 

Question  on 

df  =  4 

Faral 1  lari ty 

X^ 

Probabi 1 1 ty 

32. 

Statistical  Analysis 

3.49 

0.4795 

37. 

Linear  Programming 

8.67 

0.0700 

42. 

Simulation 

3.75 

0.4409 

47. 

Project  Selection 

6.35 

0. 1744 

52. 

Decision  Theory 

6.83 

0 . 1450 

57. 

Banking 

4.07 

0 . 3962 

62. 

Networking 

7.80 

0 . 0993 

67. 

Financial  Methods 

7.73 

0. 1018 

2. BANK 

Kruskal-Wal 11s 

Question  on 

df  =  4 

Use 

X^* 

Probabi 1 1 ty 

33. 

Statistical  Analysis 

4.06 

0.3977 

38. 

Linear  Programming 

3.56 

0.4681 

43. 

Simulation 

2.35 

0.6718 

48. 

Project  Selection 

2.73 

0.6040 

53. 

Decision  Theory 

2.68 

0.6124 

58. 

Banking 

1 . 25 

0.8700 

63. 

Networking 

2.37 

0.6689 

68. 

Financial  Methods 

7.57 

0. 1086 
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2.RA¥K 

Kruskal-Wallls 

Question  on 

df  =  4 

Benefit 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

7.96 

0.0031 

39. 

Linear  Programming 

5.61 

0.2306 

44. 

Simulation 

1.02 

0.9074 

40. 

Project  Selection 

2.47 

0.6495 

54. 

Decision  Theory 

0.B5 

0.9317 

59. 

Banking 

4 . 55 

0 . 3366 

64  . 

Networking 

1 . 40 

0.8283 

69. 

Financial  Methods 

5.60 

0.2314 

2. BANK 

Kruskal-Wallls 

Question  on 

df  =  4 

Aval labi 1 1 ty 

X=* 

Probabl 1 1 ty 

35. 

Statistical  Analysis 

5.41 

0.2482 

40. 

Linear  Programming 

7.84 

0 . 0975 

45  . 

Simulation 

2 . 82 

0 . 5887 

50. 

Project  Selection 

2.61 

0.6251 

55. 

Decision  Theory 

1 . 46 

0  8331 

60. 

Ranking 

2 . 76 

0  5088 

65  . 

Networking 

2  .  18 

0 . 7025 

70. 

Financial  Methods 

8 . 38 

0.0787 

2. RANK 

Kruskal -Wal 1  i  a 

Question  on 

df  =  4 

Opportunity 

X= 

Probabl 1 1 ty 

36. 

Statistical  An&lys ia 

4.75 

0.3140 

41  . 

Linear  Programming 

6. 10 

0. 1857 

46. 

Simulation 

3.28 

0.5116 

51  . 

Project  Selection 

2. 13 

0.7118 

56. 

Decision  Theory 

1.01 

0.9080 

61  . 

Banking 

2.21 

0.6966 

66. 

Networking 

1  74 

0  7832 

71  . 

Financial  Methods 

5  80 

0 . 2075 

3. HIGHEST  DEGREE 

Question  on 

Faml 1 lar 1 ty 

Kruskal -Wal 11s 
df  =  3 

X^ 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

1.15 

0.7640 

37. 

Linear  Programming 

2.31 

0 .5102 

42. 

Simulation 

1 .03 

0.7949 

47  . 

Project  Selection 

2. 14 

0.5434 

52. 

Decision  Theory 

6.42 

0.0928 

57. 

Banking 

1 . 30 

0 . 7079 

62. 

Networking 

2.41 

0.4015 

67. 

Financial  Methods 

2.76 

0 . 4206 
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3. HIGHEST  DEGREE 

Kruskal -Wal 11s 

Question  on 

df  =  3 

Use 

X* 

Probabi 1 1 ty 

33. 

Statistical  Analysis 

5.11 

0. 1641 

38. 

Linear  Programming 

0.78 

0.8535 

43. 

Simulation 

0.45 

0.9301 

48. 

Project  Selection 

3.65 

0.3024 

53. 

Decision  Theory 

1.55 

0.6698 

58. 

Ranking 

2.64 

0.2666 

63. 

networking 

7. 15 

0.0671 

68. 

Financial  Methods 

2 . 50 

0.4755 

3. HIGHEST  DEGREE 

Kruskal -Wal 11s 

Question  on 

df  =  3 

Benef i t 

Probabi 1 1 ty 

34. 

Statistical  Analysis 

10.54 

0.0145 

39. 

Linear  Programming 

0.56 

0.9048 

44  . 

Simulation 

0.97 

0 . 8082 

49. 

Project  Selection 

3.22 

0.3592 

54. 

Decision  Theory 

1 . 02 

0.7959 

59. 

Ranking 

2.49 

0 . 4771 

64  . 

Network i ng 

3.31 

0 . 3456 

69  . 

Financial  Methods 

2.75 

0 . 4325 

3. HIGHEST  DEGREE 

Kruskal -Wall  la 

Question  on 

df  =  3 

Aval lability 

Probabi 1 1 ty 

35. 

Statistical  Analysis 

2.64 

0 . 4504 

40. 

Linear  Programming 

0.91 

0.8235 

45. 

Simulation 

2.25 

0.5218 

50. 

Project  Selection 

2.07 

0.5585 

55. 

Decision  Theory 

0.33 

0.9540 

60. 

Ranking 

0.83 

0 . 8428 

65  . 

Networking 

1 .51 

0.6789 

70. 

Financial  Methods 

1 . 25 

0.7399 

3. HIGHEST  DEGREE 

Kruskal-Wal 11s 

Question  on 

df  =  3 

Opportunl ty 

Probabi 1 1 ty 

36. 

Statistical  Analysis 

4 . 06 

0 . 2556 

41  . 

Linear  Programming 

0.59 

0.8993 

46. 

Simulation 

0.30 

0 . 9601 

51  . 

Project  Selection 

2.43 

0 . 4889 

56. 

Decision  Theory 

1 . 38 

0.7101 

61  . 

Ranking 

2.99 

0 . 3935 

66. 

Networking 

6.66 

0.0835 

71  . 

Financial  Methods 

1 . 28 

0 . 7345 
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4. FIELD  OF  STUDY 

Kruskal -Wal 11s 

Question  on 

df  =  3 

Familiarity 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

3 . 15 

0 . 3697 

37. 

Linear  Programming 

5.76 

0 . 1239 

42. 

Simulation 

2 . 77 

0 . 4289 

47. 

Project  Selection 

4 . 93 

0  1766 

52. 

Decision  Theory 

5 . 44 

0 . 1424 

57  . 

Banking 

1  .  18 

0  7574 

62. 

Network  1 ng 

5.10 

0 . 1643 

67. 

Financial  Methods 

3.28 

0 . 3501 

4  -  FI 

ELD  OF  STUDY 

Kruskal-Wal 1  is 

Question  on 

df  =  3 

Use 

Probabl 1 1 ty 

33. 

Statistical  Analysis 

2.45 

0.4837 

38. 

Linear  Programming 

3.44 

0 . 3292 

43. 

Simulation 

2.23 

0.5271 

48. 

Project  Selection 

2.50 

0 . 4760 

53. 

Decision  Theory 

0.85 

0.8364 

58. 

Banking 

1  20 

0.7528 

63. 

Networking 

0 . 88 

0.8304 

68. 

Financial  Methods 

3.97 

0.2649 

4. FI ELD  OF  STUDY 

Kruskal-Wal lie 

Question  on 

df  =  3 

Benef 1 t 

X= 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

0.61 

0.8942 

39. 

Linear  Programming 

6.29 

0.0984 

44  . 

S 1 mul at 1 on 

1  .  42 

0.6997 

49. 

Project  Selection 

6.50 

0.0896 

54. 

Decision  Theory 

0.23 

0.9725 

59  . 

Banking 

0.96 

0 .8106 

64. 

Ne  twork 1 ng 

0.58 

0.9017 

69  . 

Financial  Methods 

3.51 

0.3189 

4. FIELD  OF  STUDY 

Kruskal-Wal 1 1  a 

Question  on 

df  =  3 

Aval  lability 

Probabl 1 1 ty 

35. 

Statistical  Analysis 

1.53 

0 . 6761 

40. 

Linear  Programming 

2.30 

0.5119 

45. 

Simulation 

0.85 

0.8377 

50. 

Project  Selection 

1 . 90 

0.5941 

55. 

Decision  Theory 

0 . 07 

0.9948 

60. 

Banking 

0.63 

0.8887 

65  . 

Networking 

2 . 35 

0 . 5027 

70. 

Financial  Methods 

1 . 27 

0 . 7361 
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4. 

FIELD  OF  STUDY 

Kruskal-Wal 11s 

Question  on 

df  =  3 

Opportunity 

Probabl 1 1 ty 

36 

Statistical  Analysis 

2 . 00 

0.5718 

41 

.  Linear  Programming 

6.31 

0.0976 

46 

Simulation 

1  .  12 

0,7719 

51 

.  Project  Selection 

2 . 58 

0 . 4805 

56 

Decision  Theory 

0.50 

0  9200 

61 

.  Ranking 

1.11 

0.7750 

66 

Networking 

0 . 97 

0.8081 

71 

.  Financial  Methods 

1 . 89 

0 . 5965 

5.SP0  TYPE 

Kruskal-Wal  11a 

Question  on 

df  =  2 

Fami 1 iar 1 ty 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

3.53 

0.1715 

37. 

Linear  Programming 

4 . 83 

0.0893 

42. 

Simulation 

1.45 

0.4836 

47. 

Project  Selection 

1 .00 

0 . 6054 

52. 

Decision  Theory 

2 . 25 

0.3241 

57. 

Ranking 

4.05 

0. 1321 

62. 

Networking 

0.96 

0.6180 

67. 

Financial  Methods 

1.11 

0 . 5739 

5.SP0  TYPE 

Kruskal-Wal 11s 

Question  on 

df  =  2 

Use 

x:* 

Probabl 1 1 ty 

33. 

Statistical  Analysis 

6.25 

0.0439 

38. 

Linear  Programming 

2 . 90 

0 . 2347 

43. 

S 1 mu 1  a  t i o  n 

4.81 

0.0901 

48. 

Project  Selection 

2.60 

0 . 2728 

53  . 

Decision  Theory 

3.05 

0.2172 

58. 

Ranking 

2.51 

0 . 2852 

63. 

Networking 

0.37 

0 . 8303 

68. 

Financial  Methods 

0. 15 

0 . 9287 

5.SP0  TYPE 

Kruskal-Wal 11s 

Question  on 

df  =  2 

Benef i t 

X= 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

0.77 

0.6798 

39. 

Linear  Programming 

3. 13 

0.2089 

44. 

Simulation 

4 . 59 

0. 1007 

49. 

Project  Selection 

1 . 86 

0.3942 

54. 

Decision  Theory 

2.42 

0.2980 

59. 

Ranking 

2.97 

0.2265 

64. 

Networking 

0.43 

0 . 8053 

69. 

Financial  Methods 

0. 17 

0 .9192 
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5.: 

SPO  TYPE 

Kruskal-Wallls 

Question  on 

df  =  2 

Av&i lability 

X' 

Probabl 1 1 ty 

35 

Statistical  Analysis 

1  .  10 

0 . 5779 

40 

Linear  Programming 

2 . 03 

0.3628 

45 

Simulation 

3.68 

0. 1588 

50 

Project  Selection 

7.98 

0.0185 

55 

.  Decision  Theory 

0 . 58 

0 . 7478 

60 

Banking 

3.74 

0. 1542 

65 

Networking 

0.43 

0 . 8066 

70 

Financial  Methods 

0.62 

0.7325 

5.: 

SPO  TYPE 

Kruskal-Wallls 

Question  on 

df  =  2 

Opportunity 

X= 

Probab 1 1 1 ty 

36 

Statistical  Analysis 

6.55 

0 . 0378 

41 

Linear  Programming 

5 . 40 

0.0674 

46 

Simulation 

1 . 85 

0 . 3961 

51 

Project  Selection 

3.62 

0 . 1637 

56 

Decision  Theory 

1 . 03 

0 . 5978 

61 

Ranking 

2.51 

0 . 2857 

66 

Networking 

0  19 

0.9094 

71 

Financial  Methods 

2 . 72 

0 . 2571 

6. JOB  POSITION 

Kruskal-Wallls 

Question  on 

df  =  3 

Faml 1  lari ty 

Probabl 1 1 ty 

32 

Statistical  Analysis 

2 . 09 

0.3519 

37 

Linear  Programming 

7.20 

0.0273 

42 

Simulation 

3 . 58 

0 . 1668 

47 

Project  Selection 

1 . 82 

0.4033 

52 

Decision  Theory 

1  .  14 

0 . 5660 

57 

Ranking 

I  .  38 

0  5026 

62 

Networking 

1 . 66 

0.4360 

67 

Financial  Methods 

3 . 62 

0. 1638 

6. JOB  POSITION 

Kruskal -Wal 1 1  s 

Question  on 

df  =  3 

Dse 

X= 

Probabl 1 1 ty 

33 

Statistical  Analysis 

6 . 08 

0.0479 

38 

Linear  Programming 

6 . 09 

0 . 0477 

43 

Simulation 

4.75 

0 . 0930 

48 

Project  Selection 

1 .78 

0.4114 

53 

Decision  Theory 

4 . 26 

0. 1 190 

58 

Ranking 

0.04 

0.9815 

63 

Networking 

1 . 05 

0.5920 

6L 

Financial  Methods 

3  89 

0 . 1433 
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6. JOB  POSITION 

Kruskal -Wal 1 is 

Question  on 

df  =  3 

Benef it 

X* 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

=  3.43 

0. 1802 

30. 

Linear  Programming 

0.81 

0.6673 

44. 

Simulation 

1.91 

0.3847 

40. 

Project  Selection 

0.  19 

0.9077 

54. 

Decision  Theory 

5.44 

0.0660 

59. 

Ranking 

0.04 

0.9815 

64. 

Networking 

2.40 

0.3009 

69. 

Financial  Methods 

3.42 

0. 1810 

6. JOB  POSITION 

Kruskal -Wal 11s 

Question  on 

df  =  3 

Aval labi 1 i ty 

X  = 

Probabl 1 i ty 

35. 

Statistical  Analysis 

12.35 

0.0021 

40. 

Linear  Programming 

5.24 

0 . 0727 

45  . 

Simulation 

5.56 

0 . 0622 

50. 

Project  Selection 

0.05 

0.9770 

55. 

Decision  Theory 

0.11 

0.9482 

60. 

Ranking 

4 . 84 

0 . 0888 

65. 

Network 1 ng 

1 .95 

0 . 3777 

70. 

Financial  Methods 

1 . 89 

0.3891 

6. JOB  POSITION 

Kruskal -Wal lls 

Question  on 

df  =  3 

Opportunity 

Probabl 1 1 ty 

36. 

Statistical  Analysis 

6.70 

0.0350 

41  . 

Linear  Programming 

0 . 16 

0.9234 

46. 

Simulation 

1.06 

0.5875 

51  . 

Project  Selection 

0 . 35 

0 . 8406 

56. 

Decision  Theory 

2.70 

0.2596 

61  . 

Ranking 

1.41 

0.4953 

66. 

Networking 

4 . 46 

0 . 1076 

71  . 

Financial  Methods 

2 . 38 

0 . 3049 

7. CUM.  EXPERIENCE 

Question  on 

Fami 1 lar 1 ty 

Kruskal -Wal lls 
df  =  4 

X=> 

Probab 1 1 1 ty 

32 

Statistical  Analysis 

3.53 

0 . 4728 

37 

Linear  Programming 

6.25 

0. 1809 

42 

Simulation 

5.84 

0.2118 

47 

Project  Selection 

5 . 30 

0 . 2577 

52 

Decision  Theory 

5.09 

0. 2780 

57 

Ranking 

4 . 88 

0.2998 

62 

.  Networking 

6 . 16 

0 . 1874 

67 

Financial  Methods 

8 . 73 

0 . 0683 
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7. CUM.  EXPERIENCE 

Kruskal -Wal 1 la 

Question  on 

df  =  4 

Use 

X= 

Probabl 1 1 ty 

33. 

Statistical  Analysis 

1.92 

0.7506 

3B. 

Linear  Programming 

4.09 

0.2880 

43. 

Simulation 

2. 17 

0.7041 

48. 

Project  Selection 

4.99 

0.2886 

S3. 

Decision  Theory 

8.52 

0.0744 

58. 

Ranking 

8.89 

0.0640 

63. 

Networking 

4  74 

0.3148 

68. 

Financial  Methods 

12.11 

0.0166 

7. CUM.  EXPERIENCE 

Kruskal -Wal 1 Is 

Question  on 

df  =  4 

Benef 1 t 

X= 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

7.70 

0. 1033 

39. 

Linear  Programming 

6.82 

0. 1458 

44  . 

Simulation 

1 . 58 

0.8120 

49. 

Project  Selection 

3.11 

0 . 5404 

54. 

Decision  Theory 

4.23 

0.3752 

59. 

Ranking 

6  81 

0. 1462 

64  . 

Networking 

4 . 97 

0.2900 

69. 

Financial  Methods 

12.39 

0.0147 

7. CUM.  EXPERIENCE 

Kruskal -Wal 1 is 

Question  on 

df  =  4 

Aval labl 1 1 ty 

X^ 

Probabl 1 1 ty 

35. 

Statistical  Analysis 

6.11 

0. 1909 

40. 

Linear  Programming 

5.70 

0.2224 

45. 

Simulation 

5.61 

0.2300 

50. 

Project  Selection 

2.08 

0.7210 

55. 

Decision  Theory 

10.79 

0.0290 

60. 

Ranking 

6.46 

0. 1674 

65. 

Networking 

10.77 

0.0293 

70. 

Financial  Methods 

11.69 

0.0198 

7. CUM.  EXPERIENCE 

Kruskal -Wal 11s 

Question  on 

df  =  4 

Opportunity 

X=* 

Probabl 1 1 ty 

36. 

Statistical  Analysis 

3 . 53 

0.4738 

41  . 

Linear  Programming 

2 . 55 

0.6355 

46  . 

Simulation 

3.00 

0.5586 

51  . 

Project  Selection 

3.78 

0.4370 

56. 

Decision  Theory 

4.80 

0.3087 

61  . 

Ranking 

6 . 57 

0. 1602 

66. 

Networking 

9.79 

0 . 0440 

71  . 

Financial  Methods 

12. 16 

0.0162 
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8. AGE  OHOUP 

Kr uska 1 - Wa 1 1 1 s 

Question  on 

di  =  4 

Faml 1 lari ty 

X* 

Probabi 1 1 ty 

32. 

Statistical  Analysis 

1.27 

0.8660 

37. 

Linear  Programming 

2.35 

0.6726 

42. 

Simulation 

2.74 

0.6023 

47. 

Project  Selection 

3.03 

0.5533 

52. 

Decision  Theory 

4.45 

0.3488 

57. 

Ranking 

2.  14 

0.7098 

62. 

Nettvorklng 

2.86 

0.5823 

67. 

Financial  Methods 

3.54 

0.4712 

8. AGE  GROUP 

Kruskal-Wal 1 is 

Question  on 

df  =  4 

Use 

X» 

Probabi 1 1 ty 

33. 

Statistical  Analysis 

7.55 

0. 1096 

38. 

Linear  Programming 

2.83 

0.5866 

43. 

Simulation 

1  .  18 

0.8812 

48. 

Project  Selection 

2.51 

0.6437 

53. 

Decision  Theory 

4.71 

0.3181 

58. 

Ranking 

2. 15 

0.7082 

63. 

Hetvvorking 

1.62 

0.8049 

68. 

Financial  Methods 

5.00 

0 . 2877 

8. AGE  GROUP 

Kruskal-Wal 11s 

Question  on 

df  =  4 

Benef 1 t 

X=» 

Probabi 1 i ty 

34  . 

Statistical  Analysis 

4.91 

0.2968 

39. 

Linear  Programming 

4.35 

0.3608 

44. 

Simulation 

2.  14 

0.7100 

49 

Project  Selection 

8.02 

0 . 0908 

54. 

Decision  Theory 

4.61 

0.3301 

59  . 

Banking 

2.08 

0.7218 

64. 

Networking 

6 . 44 

0. 1684 

69. 

Financial  Methods 

2.25 

0.6890 

8. AGE  GROUP 

Kruskal-Wal  11s 

Question  on 

df  =  4 

Aval labl 1 1 ty 

Probabi 1 1 ty 

35. 

Statistical  Analysis 

0.09 

0.9990 

40. 

Linear  Programming 

0.33 

0.9877 

45  . 

Simulation 

1.74 

0.7840 

50. 

Project  Selection 

2.00 

0.7365 

55. 

Decision  Theory 

0.55 

0.9682 

60. 

Ranking 

0.48 

0.9758 

65. 

Networking 

1.00 

0.9095 

70. 

Financial  Methods 

3.60 

0.4627 
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8. AGE  GBOUP 

Kruskal -Wal 1 is 

Question  on 

df  =  4 

Opportunity 

Probabl 1 i ty 

36. 

Statistical  Analysis 

6.11 

0. 1912 

41  . 

Linear  Programming 

5.29 

0.2584 

46. 

Simulation 

0.67 

0.9548 

51  . 

Project  Selection 

5.62 

0.2295 

56. 

Decision  Theory 

1  87 

0.7595 

61 

Banking 

1  17 

0.8831 

66. 

Networking 

4.69 

0.3210 

71  . 

Financial  Methods 

5.75 

0.2182 

9.  ACQOISITIOll  PHASE 

Kruskal -Wal 1 is 

Question  on 

df  =  3 

Fami 1 lari ty 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

3.50 

0.3216 

37. 

Linear  Programming 

7.86 

0.0489 

42. 

Simulation 

8.87 

0.0311 

47. 

Project  Selection 

1.53 

0.6754 

52. 

Decision  Theory 

4.15 

0 . 2454 

57. 

Banking 

6.58 

0.0867 

62. 

Networking 

0.97 

0.8092 

67. 

Financial  Methods 

3.43 

0 . 3303 

9. ACQUISITION  PHASE 

Kruskal -Wal 1 is 

Question  on 

df  =  3 

Use 

X=» 

Probabl 1 i ty 

33. 

Statistical  Analysis 

5.34 

0. 1485 

38. 

Linear  Programming 

5.95 

0. 1139 

43. 

Simulation 

6.11 

0. 1063 

48. 

Project  Selection 

2.00 

0.5723 

53. 

Decision  Theory 

2 . 97 

0.3962 

58. 

Banking 

7. 12 

0.0681 

63. 

Networking 

1 . 83 

0.6079 

68. 

Financial  Methods 

6.56 

0.0874 

9. ACQUISITION  PHASE 

Kruskal -Wal 1 is 

Question  on 

df  =  3 

Benef 1 t 

X’* 

Probabl 1 1 ty 

34  . 

Statistical  Analysis 

2.51 

0.4736 

39. 

Linear  Programming 

2 . 42 

0.4901 

44. 

Simulation 

0.89 

0.8282 

49. 

Project  Selection 

0.91 

0.8227 

54. 

Decision  Theory 

5.93 

0. 1149 

59. 

Banking 

8.62 

0.0348 

64. 

Networking 

1 . 30 

0.7289 

69. 

Financial  Methods 

1.30 

0.7298 
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9. ACQUISITIOtf  PHASE  Kpuakal -Wal 1 Is 


Question  on 

df  =  3 

Availabl 1 i ty 

X= 

Probabl 1 1 ty 

35. 

Statistical  Analysis 

10.57 

0.0143 

40. 

Linear  Programming 

12.35 

0.0063 

45. 

Simulation 

17.64 

0.0005 

50. 

Project  Selection 

4.23 

0.2372 

55. 

Decision  Theory 

6.70 

0.0821 

60. 

Banking 

14.30 

0.0025 

65. 

Networking 

3. 15 

0 . 3690 

70. 

Financial  Methods 

6.03 

0. 1 102 

9. ACQUISITION  PHASE 

Kruskal -Wal 11s 

Question  on 

di  =  3 

Opportunity 

X= 

Probabl 1 1 ty 

36. 

Statistical  Analysis 

3.23 

0.3574 

41  . 

Linear  Programming 

2.06 

0 . 5602 

46. 

Simulation 

1 . 55 

0.6609 

51  . 

Project  Selection 

2.00 

0.5727 

55. 

Decision  Theory 

5.50 

0. 1387 

61  . 

Banking 

3.60 

0.3082 

66. 

Networking 

1 . 20 

0.7540 

71  . 

Financial  Methods 

2.94 

0.4016 

10 

COMPUTER  FAM 

Kruskal -Wallis 

Question  on 

df  =  4 

^aml 1 iari  ty 

X“ 

Probabl 1 1 ty 

32 

.  Statistical  Analysis 

9.91 

0.0419 

37 

Linear  Programming 

10.05 

0.0396 

42 

Simulation 

11.00 

0.0265 

47 

.  Project  Selection 

4.58 

0.3330 

52 

Decision  Theory 

11.57 

0.0209 

57 

.  Banking 

6.49 

0. 1655 

62 

Networking 

10.15 

0 . 0380 

67 

Financial  Methods 

7.08 

0. 1315 

10. COMPUTER  FAM  Kruskal -Wal 1 is 


Question  on 

df  =  4 

Use 

X=» 

Probabl 1 1 ty 

33. 

Statistical  Analysis 

5.61 

0 . 2305 

58  . 

Linear  Programming 

10 . 60 

0.0315 

43. 

Simulation 

1.56 

0 .8161 

48. 

Project  Selection 

3.45 

0.4849 

53. 

Decision  Theory 

4.06 

0 . 3976 

58. 

Banking 

3.65 

0.4557 

63. 

Networking 

5.25 

0.2622 

68. 

Financial  Methods 

5.98 

0 . 2009 
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10. 

COMPUTEB  FAM 

Kruskal -Wal 11s 

Question  on 

dl  =  4 

Benef i t 

Probability 

34. 

Statistical  Analysis 

7.46 

0. 1133 

30. 

Linear  Programming 

6  23 

0. 1825 

44. 

Simulation 

2.44 

0.6546 

40. 

Project  Selection 

4  01 

0.4046 

54. 

Decision  Theory 

4. 19 

0.3809 

50. 

Banking 

1.21 

0  8772 

64. 

Networking 

2 . 55 

0 . 6353 

60. 

Financial  Methods 

3.89 

0.4210 

10. 

COMPUTEB  FAM 

Kruskal-Wal  11s 

Question  on 

df  =  4 

Aval  lability 

X=* 

Probabl 1 1 ty 

35. 

Statistical  Analysis 

3. 19 

0.5260 

40. 

Linear  Programming 

1 . 89 

0.7554 

45. 

Simulation 

3. 54 

0.4715 

50  . 

Project  Selection 

0.53 

0.9704 

55. 

Decision  Theory 

1 .02 

0.9065 

60. 

Banking 

3.46 

0 . 4847 

65. 

Networking 

4 . 96 

0.2910 

70. 

Financial  Methods 

3.36 

0 . 4989 

10. 

COMPUTEB  FAM 

Kruskal-Wall Is 

Question  on 

df  =  4 

Opportunl ty 

X=> 

Probabl 1 1 ty 

36. 

Statistical  Analysis 

6.97 

0. 1372 

41  . 

Linear  Programming 

2.34 

0.6743 

46. 

Simulation 

1 .86 

0.7619 

51  . 

Project  Selection 

3. 17 

0 . 5294 

56. 

Decision  Theory 

1 .39 

0.8451 

61  . 

Bank 1 ng 

1 . 07 

0 . 8991 

66  . 

Networking 

5.21 

0 . 2661 

71  . 

Financial  Methods 

4.25 

0.3736 

1 1  . 

COMPUTEB  USE 

Kruskal-Wall 1 s 

Question  cn 

df  =  4 

Faml 1 lar 1 ty 

X=» 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

6 . 62 

0. 1574 

37. 

Linear  Programming 

8. 13 

0.0869 

42  . 

Simulation 

5.53 

0.2375 

47. 

Project  Selection 

1 . 49 

0.8289 

52. 

Decision  Theory 

6.51 

0. 1640 

57. 

Banking 

2.70 

0.6101 

62. 

Networking 

7.07 

0 . 1323 

67. 

Financial  Methods 

7 . 46 

0. 1135 
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11 . 

COMPUTEB  DSE 

Kruskal-Wal 1 is 

Question  on 

df  =  4 

Use 

X* 

Probabl 1 1 ty 

33. 

Statistical  Analysis 

5.43 

0.2457 

38. 

Linear  Programming 

0.80 

0.0440 

43. 

Simulation 

0.76 

0.0441 

48. 

Project  Selection 

2.01 

0.7330 

53. 

Decision  Theory 

0.03 

0.9197 

58. 

Ranking 

1.36 

0.8511 

63. 

Networking 

3.72 

0.4459 

68. 

Financial  Methods 

12.03 

0.0116 

1  1  . 

COMPUTER  DSE 

Kruskal-Wallls 

Question  on 

di  =  4 

Benef 1 t 

X=« 

Prohabl 1 1 ty 

34  . 

Statistical  Analysis 

5.40 

0 . 2486 

39. 

Linear  Programming 

10. 13 

0.0382 

44. 

Simulation 

3.21 

0.5230 

49. 

Project  Selection 

6.09 

0. 1923 

54  . 

Decision  Theory 

4.50 

0.3430 

59. 

Ranking 

6.21 

0. 1841 

64  . 

Networking 

5.80 

0.2143 

69. 

Financial  Methods 

12.43 

0.0144 

11  . 

COMPUTER  DSE 

Kruskal -Wal 11s 

Question  on 

df  =  4 

Aval lability 

X=» 

Probabl 1 i ty 

35. 

Statistical  Analysis 

2.91 

0.5726 

40. 

Linear  Programming 

3.48 

0.4804 

45. 

Simulation 

6.75 

0. 1495 

50. 

Project  Selection 

0.41 

0.0812 

55. 

Decision  Theory 

1.85 

0.7638 

60. 

Ranking 

3. 10 

0.5413 

65. 

Networking 

5.50 

0.2323 

70. 

Financial  Methods 

0.01 

0.0420 

1 1  . 

COMPUTER  DSE 

Kruskal -Wal 1 i s 

Question  on 

df  =  4 

Opportunity 

X=> 

Probabl 1 1 ty 

36. 

Statistical  Analysis 

13.23 

0.0102 

41 . 

Linear  Programming 

12.02 

0.0172 

46. 

Simulation 

3.88 

0.4222 

51 . 

Project  Selection 

8.21 

0.0843 

56. 

Decision  Theory 

3.29 

0.5110 

61 . 

Ranking 

3.37 

0 . 4977 

66. 

Networking 

6.67 

0 . 1544 

71  . 

Financial  Methods 

17.43 

0.0016 
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15. 

LOGISTICS 

Kruskal-Wal 11s 

Question  on 

df  =  4 

Fami 1 iari ty 

X=' 

Probabl 1 1 ty 

32. 

Statistical  Analysis 

5.58 

0.2326 

37. 

Linear  Programming 

6.  12 

0 . 1906 

42. 

Simulation 

6.31 

0. 1774 

47. 

Project  Selection 

9.25 

0.0551 

52. 

Decision  Theory 

8.56 

0.0732 

57. 

Ranking 

10 . 72 

0.0299 

62. 

Networking 

6.85 

0. 1439 

67. 

Financial  Methods 

10.58 

0.0317 

15. 

LOGISTICS 

Kruskal-Wal 1 is 

Question  on 

df  =  4 

Use 

X* 

Probabi 1 1 ty 

33. 

Statistical  Analysis 

2.53 

0.6396 

38. 

Linear  Programming 

3.71 

0.4470 

43. 

Simulation 

4. 14 

0.3874 

48. 

Project  Selection 

5.47 

0.2424 

53. 

Decision  Theory 

7.84 

0.0975 

58. 

Ranking 

10.69 

0.0303 

63. 

Networking 

4.74 

0.3145 

68. 

Financial  Methods 

2.74 

0.6027 

15. 

LOGISTICS 

Kruskal -Wal 1 is 

Question  on 

df  =  4 

Benef 1 t 

Probabl 1 1 ty 

34. 

Statistical  Analysis 

1 . 02 

0.9065 

30. 

Linear  Programming 

10.84 

0 . 0284 

44. 

Simulation 

2.42 

0.6599 

49. 

Project  Selection 

10.31 

0.0355 

54. 

Decision  Theory 

5.79 

0.2156 

59. 

Ranking 

13.23 

0.0102 

64. 

Network 1 ng 

3.57 

0 . 4676 

69  . 

Financial  Methods 

5.48 

0.2412 

15. LOGISTICS 

Question  on 
Availability 

35.  Statistical  Analysis 

40.  Linear  Progranuning 

45.  Simulation 

50.  Project  Selection 

55.  Decision  Theory 

60.  Banking 

65.  Networking 

70.  Financial  Methods 

15. LOGISTICS 

Question  on 
Opportunity 

36.  Statistical  Analysis 

41.  Linear  Programming 

46.  Simulation 

51.  Project  Selection 

56.  Decision  Theory 

6 1 .  Banking 

66.  Networking 

71.  Financial  Methods 


Kruskal-Wal 11s 
df  =  4 


X* 

Probabl 1 i 

2.62 

0.6232 

8.36 

0.0793 

0.63 

0.0503 

5.33 

0.2555 

10.20 

0.0372 

9.21 

0.0562 

1.38 

0.8476 

4.02 

0.4034 

Kruskal-Wal 1 is 

df  =  4 

X* 

Probabi 1 1 

3.20 

0.5245 

7.02 

0. 1350 

5.74 

0.2191 

8.44 

0.0768 

4.00 

0.2880 

11.35 

0.0220 

6.58 

0. 1500 

3.76 

0.4388 

ty 


ty 


147 


Statistical  Analysis 


CONTROL 

TREATMEN’i 

QUESTION 

RESPONSE 

FREQUENCY 

X 

FREQUENCY 

X 

32. 

NONE 

2 

4.9 

1 

2.9 

FAMILIAR 

SLIGHTLY 

13 

31.7 

5 

14.7 

MODERATELY 

20 

48.6 

14 

41.2 

VERY 

S 

12.2 

8 

23.5 

COMPLETELY 

1 

2.4 

6 

17.6 

33. 

NEVER 

18 

43.9 

6 

17.6 

USE 

SELDOM 

12 

29.3 

15 

44 . 1 

OCCASIONALLY 

9 

22.0 

10 

29.4 

OFTEN 

2 

4.9 

3 

8.8 

ALWAYS 

0 

0.0 

0 

0.0 

34  . 

NEVER 

4 

9.8 

2 

5.9 

BENEFIT 

SELDOM 

16 

39.0 

1  1 

32 . 4 

OCCASIONALLY 

17 

415 

13 

38.2 

OFTEN 

4 

9.8 

8 

23.5 

ALWAYS 

0 

0.0 

0 

0.0 

35. 

NO  ANSWER 

1 

N/A 

0 

N/A 

AVAILABLE 

NONE 

10 

25.0 

3 

8.8 

SLIGHTLY 

15 

37.5 

17 

50.0 

MODERATELY 

6 

15.0 

1 

2 . 9 

VERY 

5 

12.5 

6 

17.6 

COMPLETELY 

4 

10.0 

7 

20 . 6 

36. 

NEVER 

4 

9.8 

1 

2 . 9 

OPPORTUNITY 

SELDOM 

20 

48.8 

14 

41.2 

OCCASIONALLY 

15 

36.6 

14 

41.2 

OFTEN 

1 

2.4 

5 

14.7 

ALWAYS 

1 

2 . 4 

0 

0 . 0 
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Linear 

QUESTION 

Programming  and 

RESPONSE 

Other  Math  Programing  Methods 

CONTROL  TREATMENT 

FREQUENCY  X  FREQUENCY  X 

37. 

NONE 

15 

36.6 

5 

14.7 

FAMILIAfi 

SLIGHTLY 

14 

34 . 1 

8 

23.5 

1I0DERATEL7 

8 

19.5 

1 1 

32.4 

VERY 

3 

7.3 

7 

20.6 

COMPLETELY 

1 

2.4 

3 

8.8 

38. 

NEVER 

26 

63.4 

17 

50.0 

USE 

SELDOM 

13 

31.7 

9 

26 . 5 

OCCASIONALLY 

1 

2 . 4 

8 

23.5 

OFTEN 

1 

2,4 

0 

0.0 

ALWAYS 

0 

0.0 

0 

0 . 0 

39. 

NO  ANSWER 

0 

N/A 

1 

N/A 

BENEFIT 

NEVER 

19 

46.3 

8 

24 . 2 

SELDOM 

8 

19 . 5 

13 

39.4 

OCCASIONALLY 

12 

29 . 3 

1 1 

33 . 3 

OFTEN 

2 

4 . 9 

1 

3 . 0 

ALWAYS 

0 

0.0 

0 

0 . 0 

40. 

NO  ANSWER 

0 

N/A 

2 

N/A 

AVAILABLE 

NONE 

22 

53.7 

14 

43.8 

SLIGHTLY 

a 

19.5 

9 

28 . 1 

MODERATELY 

6 

14.6 

3 

9.4 

VERY 

2 

4.0 

4 

12.5 

COMPLETELY 

3 

7.3 

2 

6.3 

41  . 

NO  ANSWER 

0 

N/A 

1 

N/A 

1  OPPORTUNITY  NEVER 

18 

43 . 9 

10 

30.3 

SELDOM 

12 

29.3 

13 

39.4 

OCCASIONALLY 

8 

19.5 

8 

24 . 2 

OFTEN 

3 

7 . 3 

2 

6 . 1 

ALWAYS 

0 

0.0 

0 

0 . 0 

149 


Simulation 


CONTROL 

TREATMENT 

QUESTION 

RESPONSE 

FREQUENCY 

X 

FREQUENCY 

X 

42. 

NONE 

7 

17. 1 

12 

35.3 

KAMI LI AB 

SLIGHTLY 

14 

34  1 

16 

47 . 1 

MODERATELY 

1  1 

26.8 

2 

5.9 

VERY 

9 

22  0 

2 

5.9 

COMPLETELY 

0 

0.0 

2 

5 . 9 

43. 

NEVER 

16 

39.0 

3 

8.8 

USE 

SELDOM 

15 

36 . 6 

19 

55 . 9 

OCCASIONALLY 

6 

14.6 

6 

17.6 

OFTEN 

4 

9  8 

4 

11.8 

ALWAYS 

0 

0.0 

2 

5.9 

44  . 

NO  ANSWER 

0 

N/A 

1 

N/A 

BENEFIT 

NEVER 

9 

22.0 

9 

27.3 

SELDOM 

12 

29 . 3 

14 

42 . 4 

OCCASIONALLY 

12 

29.3 

1 

3 . 0 

OFTEN 

6 

14 . 6 

5 

15.2 

ALWAYS 

2 

4.9 

4 

12  .  1 

45. 

NONE 

13 

31 .7 

5 

14.7 

AVAILABLE 

SLIGHTLY 

15 

36.6 

18 

52 . 9 

MODERATELY 

5 

12.2 

8 

23.5 

VERY 

4 

9.8 

1 

2.9 

COMPLETELY 

4 

9.8 

2 

5  9 

46. 

NEVER 

10 

24.4 

7 

20 . 6 

OPPORTUNITY 

SELDOM 

14 

34 . 1 

16 

47. 1 

OCCASIONALLY 

12 

29.3 

9 

26.5 

OFTEN 

4 

9.8 

0 

0 . 0 

ALWAYS 

1 

2.4 

2 

5 . 9 

1 


150 


Project  Selection 


CONTROL 

TREATMENT 

QUESTION 

RESPONSE 

FREQUENCY 

X 

FREQUENCY 

X 

47. 

NO  ANSWER 

1 

N/A 

0 

N/A 

FAMILIAR 

NONE 

24 

60.0 

24 

70.6 

SLIGHTLY 

7 

17.5 

7 

20.6 

MODERATELY 

5 

12 . 5 

2 

5.9 

VERY 

3 

7.5 

0 

0 . 0 

COMPLETELY 

1 

2.5 

1 

2.0 

48. 

NO  ANSWER 

1 

N/A 

0 

N/A 

USE 

NEVER 

30 

75.0 

24 

70.6 

SELDOM 

6 

15.0 

7 

20 . 6 

OCCASIONALLY 

4 

10.0 

2 

5.9 

OFTEN 

0 

0.0 

0 

0 . 0 

ALWAYS 

0 

0.0 

1 

2.9 

49. 

NO  ANSWER 

1 

N/A 

0 

N/A 

BENEFIT 

NEVER 

25 

62.5 

15 

44 . 1 

SELDOM 

6 

15.0 

13 

38.2 

OCCASIONALLY 

7 

17.5 

5 

14.7 

OFTEN 

2 

5.0 

0 

0.0 

ALWAYS 

0 

0 . 0 

1 

2 . 9 

50. 

NONE 

31 

75.6 

18 

52.9 

AVAILABLE 

SLIGHTLY 

5 

12.2 

10 

29.4 

MODERATELY 

4 

9.8 

3 

8.8 

VERY 

1 

2.4 

2 

5.9 

COMPLETELY 

0 

0.0 

1 

2 . 9 

51  . 

NEVER 

25 

61.0 

16 

47 . 1 

OPPORTUNITY 

SELDOM 

9 

22.0 

1 1 

32 . 4 

OCCASIONALLY 

5 

12.2 

5 

14.7 

OFTEN 

2 

4.9 

1 

2 . 9 

ALWAYS 

0 

0 . 0 

1 

2 . 9 
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Frequency  of  Besponaes 


Decision  Theory 


CONTROL 

TREATMENT 

gUESTIOH 

RESPONSE 

FREQUENCY 

Z 

FREQUENCY 

X 

52  . 

NONE 

1 

2.4 

0 

0 . 0 

FAMILIAR 

SLIGHTLY 

13 

31.7 

6 

17.6 

MODERATELY 

14 

34. 1 

17 

50.0 

VERY 

12 

29.3 

8 

23.5 

COMPLETELY 

1 

2.4 

3 

8.8 

53. 

NO  ANSWER 

1 

N/A 

0 

N/A 

USE 

NEVER 

14 

35.0 

7 

20.6 

SELDOM 

14 

35.0 

17 

50.0 

OCCASIONALLY 

7 

17.5 

6 

17.6 

OFTEN 

5 

12 . 5 

4 

11.8 

ALWAYS 

0 

0.0 

0 

0.0 

54. 

NO  ANSWER 

1 

N/A 

0 

N/A 

BENEFIT 

NEVER 

9 

22.5 

6 

17.6 

SELDOM 

12 

30 . 0 

1  1 

32 . 4 

OCCASIONALLY 

1  1 

27 . 5 

14 

41.2 

OFTEN 

8 

20.0 

2 

5.9 

ALWAYS 

0 

0.0 

1 

2.9 

55. 

NONE 

15 

36.6 

9 

26.5 

AVAILABLE 

SLIGHTLY 

1  ] 

26.8 

13 

38.2 

MODERATELY 

6 

14.6 

8 

23.5 

VERY 

8 

10.5 

2 

5 . 9 

COMPLETELY 

1 

2.4 

2 

5.9 

56  . 

NEVER 

1  I 

26.0 

6 

17 . 6 

OPPORTUNITY 

SELDOM 

14 

34 . 1 

16 

47  1 

OCCASIONALLY 

13 

31.7 

8 

23.5 

OFTEN 

3 

7.3 

3 

8 . 8 

ALWAYS 

0 

0.0 

1 

2  9 
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Ranking 


CONTROL 

TREATMENT 

QUESTION 

RESPONSE 

FREQUENCY 

X 

FREQUENCY 

X 

57. 

NONE 

10 

24.4 

2 

5.9 

FAMILIAR 

SLIGHTLY 

8 

19.5 

7 

20.6 

MODERATELY 

13 

31.7 

17 

50.0 

VERY 

7 

17 . 1 

6 

17.6 

COMPLETELY 

3 

7.3 

2 

5.9 

58. 

NO  ANSWER 

1 

N/A 

1 

N/A 

USE 

NEVER 

13 

32 . 5 

7 

21.2 

SELDOM 

13 

32 . 5 

13 

39 . 4 

OCCASIONALLY 

8 

20.0 

5 

15.2 

OFTEN 

6 

15.0 

8 

24 . 2 

ALWAYS 

0 

0.0 

0 

0.0 

59. 

NO  ANSWER 

0 

N/A 

1 

N/A 

BENEFIT 

NEVER 

9 

22 . 0 

4 

12. 1 

SELDOM 

1 1 

26.8 

10 

30 . 3 

OCCASIONALLY 

13 

31 .7 

1  1 

33.3 

OFTEN 

8 

19 . 5 

7 

21.2 

ALWAYS 

0 

0.0 

1 

3.0 

60. 

NO  ANSWER 

0 

N/A 

1 

N/A 

AVAILABLE 

NONE 

19 

46.3 

1  1 

33.3 

SLIGHTLY 

8 

19 . 5 

1 1 

33.3 

MODERATELY 

8 

19.5 

3 

9 . 1 

VERY 

3 

7.3 

5 

15.2 

COMPLETELY 

3 

7.3 

3 

9. 1 

61  . 

NO  ANSWER 

0 

0 . 0 

1 

N/A 

OPPORTUNITY 

NEVER 

12 

29.3 

5 

15.2 

SELDOM 

1  1 

26.8 

13 

39 . 4 

OCCASIONALLY 

10 

24 . 4 

7 

21.2 

OFTEN 

8 

19.5 

7 

21.2 

ALWAYS 

0 

0 . 0 

1 

3.0 
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Metwopking 


CONTROL 

TREATMENT 

QUESTIOH 

RESPONSE 

FREQUENCY 

X 

FREQUENCY 

X 

62. 

NONE 

0 

0.0 

0 

0.0 

FAIIILIAB 

SLIGHTLY 

4 

9.8 

2 

5.9 

MODERATELY 

11 

26.8 

9 

26.5 

VERY 

20 

48.8 

14 

412 

COMPLETELY 

6 

14.6 

9 

26  5 

63. 

NEVER 

4 

9.8 

1 

2.9 

USE 

SELDOM 

7 

17. 1 

9 

26 . 5 

OCCAS [ONALLY 

12 

29 . 3 

1  1 

32.4 

OFTEN 

14 

34. 1 

10 

29.4 

ALWAYS 

4 

9.8 

3 

8.8 

64. 

NEVER 

3 

7.3 

1 

2 . 9 

BElfEFIT 

SELDOM 

6 

14 . 6 

5 

14.7 

OCCASIONALLY 

1  1 

26.8 

16 

47. 1 

OFTEN 

16 

39 . 0 

10 

29 . 4 

ALWAYS 

5 

12.2 

2 

5.9 

65  . 

NONE 

4 

9.8 

3 

8.8 

AVAILABLE 

SLIGHTLY 

9 

22.0 

9 

26 . 5 

MODERATELY 

5 

12.2 

7 

20.6 

VERY 

16 

39.0 

11 

32  4 

COMPLETELY 

7 

17.  1 

4 

11.8 

66  . 

NEVER 

4 

9.8 

2 

5.9 

OPPORTUNITY 

SELDOM 

7 

17. 1 

9 

26.5 

OCCASIONALLY 

12 

29.3 

1 1 

32.4 

OFTEN 

15 

36.6 

9 

26.5 

ALWAYS 

3 

7.3 

3 

8.8 
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Financial  Methods 


CONTROL  TREATMENT 

QUESTION  RESPONSE  FREQUENCY  X  FREQUENCY  X 

C7.  NO  ANSWER  1  N/A  0  K/A 

FAMILIAR  NONE  5  12.5  0  0.0 

SLIGHTLY  7  17.5  6  17.6 

MODERATELY  17  42.5  9  26.5 

VERY  10  25.0  10  29.4 

COMPLETELY  1  2.5  9  26.5 

68.  NO  ANSWER  2  N/A  0  N/A 

USE  NEVER  13  33.3  2  5.9 

SELDOM  9  23.1  10  29.4 

OCCASIONALLY  11  28.2  8  23.5 

OFTEN  5  12.8  9  26.5 

ALWAYS  1  2.6  5  14.7 


69. 

NO  ANSWER 

1 

N/A 

0 

N/A 

BENEFIT 

NEVER 

10 

25 . 0 

2 

5 . 9 

SELDOM 

7 

17.5 

7 

20.6 

OCCASIONALLY 

13 

32.5 

8 

23.5 

OFTEN 

8 

20.0 

1  1 

32.4 

ALWAYS 

2 

5.0 

6 

17.6 

70. 

NO  ANSWER 

2 

N/A 

0 

N/A 

AVAILABLE 

NONE 

12 

30.8 

3 

8.8 

SLIGHTLY 

12 

30.8 

10 

29 . 4 

MODERATELY 

5 

12.8 

3 

8.8 

VERY 

6 

15.4 

1 1 

32 . 4 

COMPLETELY 

4 

10 . 3 

7 

20 . 6 

71  . 

NO  ANSWER 

2 

N/A 

1 

N/A 

OPPORTUNITY 

NEVER 

13 

33 . 3 

2 

6. 1 

SELDOM 

8 

20.5 

8 

24 . 2 

OCCASIONALLY 

1 1 

28 . 2 

7 

21.2 

OFTEN 

5 

12.8 

12 

36 . 4 

ALWAYS 

2 

5. 1 

4 

12. 1 

Appendix  M:  Data  from  SY5-400  Survey 


Haw  survey  data  from  the  test  conducted  in  the 
Intermediate  Program  Management  (SYS-400)  Professional 
Continuing  Education  (PCE)  course  at  Wright-Patterson 
AFB,  OH. 
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Baw  Survey  Data  -  SYS-400  8dA  -  Control  Oroup 
If  Is  the  synbol  for  Missing  data  -  Vo  Answer 
Q4II  to  Q4IV  are  the  wultlple  answers,  if  any,  to  Q4 
Q9II  to  Q9IV  are  the  multiple  answers,  If  any,  to  Q9 


Q  Q 

Q  4  Q  Q  9  Q 


0 

4 
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4 
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Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

B 

I 

Q 

Q 

Q 

Q 

I 

I 

I 

Q 

Q 

Q 

Q 

Q 

I 

I 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

S 

D 
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M 
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3 

3 

2 
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4 
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13 

13 

4 
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15 
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4 

1 

2 

17 

17 

2 

2 

2 

1 

3 

M 
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18 
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4 
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2 
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4 

2 

1 

M 

M 
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4 

4 

4 

2 

4 

1 

4 

2 

3 

2 

3 

2 

3 

19 

19 

2 

2 

2 

1 

M 

M 

M 

1 

2 

5 

3 

3 

M 

M 

M 

5 

3 

3 

3 

3 

2 

3 
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3 
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2 
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1 
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22 

22 
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3 
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3 

M 

M 

M 

5 

3 

4 

4 

2 

2 

3 

1 

1 

3 

3 

1 

1 

1 

3 

OQQQQQQQQQQQQQQQQ 

B2222233333333334 

S5878901234567890 

12331313333423123 
24S45554322M12114 
31111111332221111 
4532425333  7  323223 
53535553312121111 
63535243322221111 
73433442213131131 
84544144413135231 
93211  1  12323524245 


QQQQQQQQQQQQQQQ 

444444444555555 

123456789012345 

224544323124234 
3322  1  1HMM423HM4 
123322111113111 
232232222214322 
111111111113331 
121  123333333332 
221212111112231 
243323533145221 
231151111113111 
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OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B3222233333333334444444444555555 

S56780012345678Q01234S678B012345 


10  1414211211222111111111111112111 

11  3223122433232111133322111113111 

12  3313334334234232244S44434334444 

13  4444324322523225243353322224224 

14  3434122213231  1333223221  1  1  1  12232 

15  1222111212222112122221111112122 

16  1212231112221111111111111112221 

17  2233311214421111121322111112143 

18  4244444333333233332323322334233 

19  4414212222112211133433111112111 

20  5M255454435534344222421  1  1  1  14434 

21  3323233512222122232333111113232 

22  1211214422323223242222322214222 


QQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B55556666666666777777777788 

S87890123456789012345678901 

1243433454553225232353IIUMIfll 

233M354544443H4UM1I1I1I1IMM1IMMM 

31  1  1  1  1  1333231  1  1  1  llfMMlfllMMMlfM 
423222252222422221fUMMIIUUlfliM 
S3S23125J3123231  IMMMliUMyUMM 
62444244555442322MMMHIfMMllMlf 
721  1  1  1  13442321323MMMMM1IMU1IM 
82444  145555554535yMlilfMMliliMU 
9  132333434444232  IMUlfVMlflflllfU 

10  1  1  1  1  1  13332321  1  1  IHMIlllllUMMUM 

11  1322223222231  1  1  llfMMlfUlfllMlIM 

12  4334234444343433MUlfyMMHMUy 

13  2433534224243353yyyyyyyyyy 

14  2  I  22224455533444yyyyyyyyyy 

15  221  1  1  1223331  1  1  1  lyyyyyyyyyy 

16  22  1  1  1  1221  1  121  1  1  lyyyyyyyyyy 

17  3  1  1  1  1  13  1  13  1  1  1  1  1  lyyyyyyyyyy 

18  3444'44444443334yyyyyyyyyy 

19  1322.  i444443332iyyyyyyyyyy 

20  3433434444444444yyyyyyyyyy 

21  3223  i33334322222yyyyyyyyyy 

22  2422224333244444yyyyyyyyyy 
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I 


ID  Written  Beapongea 

1.  S.  Hybrid  SPO,  Staffing  Function,  Product  Dlvlalon, 

Strategic  Planning  Dlvlalon 
6.  Dlvlalon  Chief,  Several  amall  atudlea 
72.  Lotua  1-2-3,  Engineering  Analyala,  apreadaheeta ,  databaae 
prograM&lng 

3.  S.  2816  in  4050TW 

6.  Dlvlalon  chief.  Fit  Teat  Engineering 

4.  6.  Director,  Acqulaltlon  Support 

10.  1.  Foreign  Technology  Dlvlalon 

5.  Directorate  of  Intelligence 

6.  Director  of  Intelligence 

9.  H/A  -  aupport  all  ASD  regardleaa  of  phaae 

11.  1.  Teat  Center 

5.  Teat  Center 

6.  Flight  Teat  Engineer 

12.  1.  HQ  SAC/XPPM  Program  Management  A  Analyala  Dlvlalon 

5.  PPBS  analyala  A  SAC  board  Structure 

6.  Board  atructure  panel  chairman  A  analyat 

13.  1.  Balllatic  Mlaaile  Office 

14.  1.  Armament  Dlvlalon 

4.  Education 

5.  HQ  AWS 

6.  HQ  AWS  Acqulaltlon  Project  Officer 

16.  6.  Deputy  director  for  conflguatlon  management 

17.  6.  Deputy  Chief  Flight  Syatem  Engineering  Dlvlalon 

18.  1.  FTD 

5.  Intelligence 
20.  1.  HQ  DSAF 

5.  H/A 

6.  PEM 

22.  1.  Balllatic  Mlaaile  Dlvlalon 

6.  Dlvlalon  Chief  In  Program  Control 
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Baw  Survey  Data  -  S7S-400  BOB  -  Control  Oroup 
If  is  the  ayabol  for  Missing  data  -  Vo  Answer 
Q4II  to  Q4IV  are  the  multiple  answers,  if  any,  to  Q4 
Q9II  to  QOIV  are  the  multiple  answers.  If  any,  to  QO 


Q 

Q  4  Q 

0  4  14 

B  IQQQQIIIQQQQQ 
S  D1234IIVS6780 

1  12221MHM24123 

2  24321MMM34543 

3  32321HMM22332 

4  42223MMM24423 

5  S4111MMM34414 

6  6422123H22424 

7  723223MM34331 

8  82322MMM14341 

0  92323MMM24243 

10102323MMM12134 
11  11  22223MM23333 
12122123MHM22524 

13  13  4413MMM34551 

14  14  2332MMM34442 

15  15  4322MMH24433 

16  16  23213MM14344 

17  17  42213MM12432 

18  18  251  1HHM34451 

19  19  4123MMM34433 


Q 

Q  9  Q 

9I9QQQQQQQQQQQQQQQ 

III111111111122222 

IIV012345678901234 

MMM341132411221411 
MMM424212211121211 
MMM223S31512431  135 
MHM223323422232422 
MMM244443224242424 
MMH42444441  1  132513 
234554443423421444 
234114444423551424 
MMM32541  11  14251412 
MMM324543S12232525 
MMM44331  1213221213 
MMM543443345233432 
234434444444143331 
3MM334432422232332 
MMM554444553334444 
MMM33342232321  1314 
MMM333423223242224 
23M333453533542325 
MMM553443223253133 


OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 
B2222233333333  3.  34444444444555555 
S5678901234567890123456789012345 

13212222223121111111111111113111 
21111111211111111122233111112211 
34122312322221111144455111114113 
4234222232323222224343441  1  124223 
53434333212122131311413114141141 
6331  13343I3332122442423213I  12444 
72343321313133141441212212123121 
82432211111221111111112111112122 
91111115323421111122222111113444 

10  3535423213131131311323113222142 

11  1111111211112111121121211112111 
12  2433234212222222221111211114442 
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OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B2222233333333334444444444555S55 

S5678g012345678Q01234S678Q012345 

13  2433113211331111144444111112222 

14  3443333334432232322323211223343 

15  4433333223222232232313113134224 

16  2312233323133231322212222123331 

17  3424434333222  1  1  1  1312121  1  1  123332 

18  4523433312411115131351111113225 

19  3323212332334333332222433334333 


OQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B55556666666666777777777788 

S67890123456789012345678901 

11211113111131111II1I1III1II11IMIIU 
2  13333322222lfUMllMlflfMlfMMllllllll 
311  1  1  1  14444422222UM11MUUMMMU 
4354334S444433333IIMliMMUlfMMM 
541  141431313314131llllilllfMMllMll 
64144145444444454554241iMllMM 
72523324345431313MIIMMUM1IUIIM 
8222222222221  1  1  1  IMMMMMMMMMM 
932222245454355S5M1IMMII1I1IU1IU 

10  31  12124453322222lililillMMIillllli 

11  12121243S443334344444MMUMM 

12  33231  14444433323MlilflllllfUMllM 

13  21  1  1  1  1333231  1  1  1  lUHMMMUlillMli 

14  33443444444332231iHMlflfUUMMU 

15  2333434334322222MHMMHMMUIIH 

16  3222  123332333323MMllliHMMIiUM 

17  1333334444444442MUyilMIIMUMy 

18  1312513125131  i4iyyyyyyyyyy 

19  3333334333333333yyyyyyyyyy 
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ID  Ifrltten  BesDoneea 

1.  6.  Engineer 

2.  1.  A  Test  Center 

5.  Test  Center 

6.  Technology  Director 

4.  6.  Teat  Manager 

5.  1.  HQ  AFCC 

5.  MAJCOM  HQ 

6.  STSTO 

6.  1.  Ballistic  Systems  Division 
72.  Exception  Reporting 

7.  5.  4050  Teat  Wing 

6.  Division  Chief,  CoBBunlcatlons  A  Computers 

8.  6.  Deputy  Director 

9.  6.  Program  Control 

11.  72.  Database  applications;  trends.  Item  tracking,  historical 

data,  status  of  accomplishments 

12.  4.  Public  Administration 

13.  1.  HQ  MAC/KPQS 

5.  Operational  Requirements  Director 

6.  Program  monitor  for  using  command.  Interface  with  single 
progam  manager 

14.  S.  Test  Wing 

6.  Program  Manager  Supervisor 

15.  1.  HQ  SAC/XR 
6.  Oaer  SYSTO 

16.  6.  Deputy  Program  Manager 

17  1.  AFTAC 

18.  5.  4050  Test  Wing 

6.  Branch  Chief 
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Baw  Survey  Data  -  S7S-400  80C  -  Treataent  Group 
M  is  the  symbol  for  Missing  data  -  llo  Answer 
Q4II  to  Q4IV  are  the  multiple  answers,  if  any,  to  Q4 
QOII  to  Q9IV  are  the  multiple  answers,  if  any,  to  Q9 


Q  Q 

Q  4  Q  Q  9  Q 

0  414  9I9QQQQQQQQQQQQQQQ 

B  IQQQQIIIQQQQQIII111111111122222 
S  D1234IIV56789IIV012345678901234 

1  1221  1MMM22443MMM34443332121  1422 

2  243213MM1424123M333444533444433 

3  341223MM331223MM332121311221111 

4  42122MMM23533MMM3233331  13122321 

5  54523MMM34553MMM3211  13331233133 

6  62321MMM22434MMM333324244224324 

7  722223MM12322MMM334544234223421 

8  821213MM12523MMM225443323323333 

9  94321MMM13232MMM344443414442314 
10102223MMM24422MMM3454  1  131  1251  1  1  1 
111125134MM34441234334444133233232 
12124S34MMM3453MMMM332223113232121 
131321223MM12412MMM424443223142433 

14  14  2444MMM24131234324312522421523 

15  15  42213MM32532MMM325443423232224 

16  16  22213MM13341234224424313231143 

17  17  2221MMM34S21MMM544444332234234 

18  18  3221MMM22424MMM423443324223334 
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OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B32322333333333344444444445S555S 

85678001234567880123436789012345 

14  1332422333531  1222222221  1  1  1  12232 

15  3324444534243231441322312324451 

16  233312422352212413124121  14121  14 

17  3424322434233232342222212234232 

18  3323322322122121242212211113222 


OQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B55556666666666777777777788 

867800123456780012345678001 

12222223233333344MlllillMMMMMy 
24344334444444444MUMMIiyilUlflf 
3IlllllllllllllllMlfMMyifUMMU 
432i2i254444433i3yyyyyyyyyy 
53322224444444444yyyyyyyyyy 
6131  1  1  131  1  1  131  1  1  lyyyyyyyyyy 
72233234434432233yyyyyyyyyy 
8222222322222222yyyyyyyyyyy 
03544444332344444yyyyyyyyyy 

10  2333333333355555yyyyyyyyyy 

11  3445444444343344yyyyyyyyyy 

12  2322224332332222yyyyyyyyyy 

13  2323524445343353yyyyyyyyyy 

14  22y4233334222322yyyyyyMyyy 

15  5444  14324  i455524yyyyyyyyyy 
I6i323422223244544yyyyyyyyyy 

17  3323235333344444yyyyyyyyyy 

18  22121  i3232232322yyyyyyyyyy 
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ID  Written  BeBPonacs 

2.  1.  Standard  System  Center 

6.  Program  Support  on  matrix  basis 

3 .  1 .  APOTEC 

5.  Test  Organization 
5 .  1 .  MSD 

5.  Knglne/ATF 

11.  4.  Logistics  Management 

5.  Logistics  Stall 

6.  S.E.  Acqulstion  Manager 

12.  1.  HQ  AFSC 

4.  English  and  History 

5.  HQ  Organization 

6.  ILS  Manager 

9.  H/A  -  I  don’t  have  a  program 
15.  1.  Human  Systems  Division 

5.  ADPO  (Advanced  Development  Program  Office) 

6.  ADPO  Director 

17.  5.  Staff  Office 

6.  Program  Director  for  Avionics  Standardization 
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Baw  Survey  Data  -  SYS-400  fi9D  -  Treatnent  Group 
If  Is  the  symbol  for  Missing  data  -  Vo  Ansiver 
Q4II  to  Q4IV  are  the  multiple  answers.  If  any,  to  Q4 
QGII  to  Q9IV  are  the  multiple  answers.  If  any,  to  Q9 


Q  Q 
Q  4  Q  Q  0  Q 

0  414  9I9QQQQQQQQQQQQQQQ 
B  IQQQQIIIQQQ0QIII111111111122222 
S  D1234IIV56789IIV012345678901234 


1  12223MMM3 

2  22221MMM2 

3  32123MMM2 

4  44113MMM3 

5  522213MM2 

6  64211MMM3 

7  7  2  5  1  1  M  M  M  1 

8  82212MMM1 

9  923213MM1 

10  10  45223MM3 

11  11422123M3 

12  12  2  5  2  1  M  M  M  1 
131342213MM3 

14  14  4223MMM2 

15  1S22234MM2 

16  16  4123MMM2 

17  17  2  1  2  1  3  M  M  2 


452  1MMM335 
4323MMM334 
25334MM344 
452134M454 
2323MMM334 
443SMMM432 
253234M323 
412234M332 
2344MMM4  15 
45S5MMM533 
43312MMS54 
4533MMM1  14 
243134M434 
4324MMH453 
2223MMM334 
3521234555 
3422MMM553 


213422352324 
443223221433 
435324343433 
332222242223 
443424452525 
342112121242 
333223232322 
331112111311 
555523222524 
333433332334 
443414121111 
432212221322 
433224142322 
422423443424 
444234234443 
555555555555 
3331431  14421 


OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B2222233333333334444444444555555 

S5678901234567890123456789012345 

1  1  121255544445332352222533235221 
24324323323221111131222111113232 
32444344323232222232323212133332 
4232433453323533225222221  1  1  14221 
534454521131III31111311212113132 
64221432322533123232253212322233 
72343323212122121122322212122232 
8412312122227321123221231111311  1 
93445422544234221244413311114231 

10  3333333423424234242342323224443 

11  4244444544545335355555555555435 

12  3544322322222111132222111113222 

13  4323322322222111221111111113333 

14  4444334433531  1MMM213M31  1323331  1 
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OQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ 

B2222233333333334444444444555555 

85678901234567890123456786012345 

15  3425444211221111121222222223222 

16  5555555321533215332153311115114 

17  3435542513443111151111111113111 


OQQQQQQOQQQOQQOOQQQQOOQQOQQ 

B55SS6666666666777777777788 

S67890123456789012345678901 

12421  1453312S5555IIMMllllUlflfMM 
23443334334332222IIMMIIMIIUMMU 
33444444444443333MllMMMlfMllMM 
413  2  31  15232154433MMMIIMMMMU11 
52222  1  23222243444II1IMMIIIIMUMM 
62323234334333443MI(U1C1IMMMMI< 
72343223333334444IIUUUUU1IUMU 
8131  1  1  14333221  1  1  lUUMlIIIMMUMM 
92322  1  254444S1322lflfMI(tfHMMMM 

10  43344454444534443343333433 

11  3543545555554454UyiiMMllMMUM 

12  2333224332322222UMMIiMUUMMM 

13  2312124222222222MMUMMMMUUy 

14  4  1  iyyy444S454455yyyyyyyyyy 

15  2322222222222222yyyyyyyyyy 

16  i434555533535553yyyyyyyyyM 

17  1  1  1  1  I  1555555  ':  555yyyyyyyyyy 
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ID  Written  BesDonaea 

1.  5.  Laboratory 

6.  Dep.  Director  Program  Control 

2.  6.  Lead  Syatema  Engineer 

4.  1.  HQ  AFCC  Scott  AFB,  IL 

5.  HQAFCC,  Scott  AFB,  IL 

6.  STSTO/Actlon  Officer 

6.  1.  HQ  AFSC 

5.  lot  preaently  In  SPO 

6,  Inapector 

9.  Hot  Applicable 

8.  6.  Dlvlalon  Chief,  ISanuf acturlng/QA 

10.  1.  AFFTC 

5.  Bange  I&M  Developntent 

6.  Branch  Chief 
0.  All  of  above 

72.  Coat  ccnatructlon  modela  for  aoftware  development  cost 
estimation . 

77.  Bisk  Asseament  Models  for  Software  Development 

11.  1.  Human  System  Division 
5.  Laboratory 

6  Project  Besearch  Englneer/Solentlf Ic  Analyst 

12.  6.  Flight  Test  Manager 

13.  1.  MAC 

5.  MAC  HQ  -  Basket  Div 

14.  1.  BSD 

6.  Program  Control  Analyst 

15.  4.  Political  Science 
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exposing  an  Air  Force  system  acquisition  midlevel  program 
manager  to  a  computer  aided  quantitative  decision  support 
technique  in  an  academic  environment,  affected  the  frequency 
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A  quantitative  decision  support  technique,  decision  tree 
analysis,  was  taught  in  the  Intermediate  Program  Management 
(SYS-400)  Professional  Continuing  Education  (PCE)  course  at 
Wright-Patterson  AFB,  OH.  Before  this  research  began,  the 
classes  were  doing  the  decision  tree  calculations  without 
the  aid  of  a  computer.  This  study  focused  on  whether 
modifying  the  curriculum  to  teach  the  use  of  PC  based 
decision  support  system  (DSS)  decision  tree  computations  to 
program  managers  would  affect  the  frequency  they  used 
quantitative  techniques  in  addressing  actual  program  and 
project  problems.  The  experiment  was  a  posttest-only  quasi¬ 
experiment  design  with  nonequivalent  groups.  This  study  did 
not  try  to  discover  if  other  teaching  techniques  or  other 
situations  will  alter  the  frequency  of  quantitative 
technique  use.  ^  . . 

Three  months  after  course  completion,  an  after-course 
survey  was  sent  to  each  course  graduate.  The  survey 
measured  the  graduate's  familiarity  and  frequency  of  using 
several  quantitative  decision  support  techniques  to  measure 
the  effect  of  the  classroom  instruction.  Demographic  and 
responsibility,  information  were  collected.  This 
information  was  used  to  identify  possible  effects,  other 
than  the  treatment  effect,  obscuring  the  analysis. 

This  study  measured  an  increase  (in  the  treatment  group) 
of  some  of  the  technique  familiarity,  use,  computerization 
benefit,  computer  program  availability  and  use  opportunity. 
The  majority  of  program  managers  are  familiar  with  several 
quantitative  decision  support  techniques.  However,  very  few 
managers  use  those  techniques  regularly. 
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